Dirt Cheap Oracle steps #3 & #4 of 6: Install Oracle XE and APEX 3.0

[Previous]

Ok kids, play time is over, it’s time to install Oracle, upgrade Apex and then turn out the lights for a good night’s sleep. After double-checking that they haven’t released 11g XE yet :(, I download Oracle XE 10gR2, as well as Apex 3.0 for good measure (no Apex 4.0 yet either…).

Download Oracle XE 10g Release 2 (10.2.0.1) for Linux x86. I chose the RPM version.

Download Oracle Application Express. I chose 3.0.1.

Did some reading, and learned a lot about configuring Linux, but in the end it was a frightfully simple matter to install the RPM using the RPM [Wikipedia:RPM] command (who would have guessed), it installed without a hitch and told me to run the configure command, which I did, and that went fine as well. So now I’ve got Oracle running.

Apex as installed along with Oracle XE works a bit differently to what I’m used to (I’m running 3.0 on Oracle 10GR2 Enterprise Edition on my WinXP box for experimental purposes), it serves as the administration GUI to the database (no Enterprise Manager, unfortunately) as well as the application development environment. The database (Oracle XE), being free, is limited to 4GB of user data, and will only use 1GB of RAM (which is fine, that’s all I’m giving it anyway). To help ensure I don’t go over 4GB I’ll keep my large media files on the file system instead of in the database, and I’ll design my applications with automated cleanup routines for tables that grow a lot. So far my applications on my experimental 10G database only use 1GB, and that’s including a whole lot of BLOBs, so I don’t think I’ll have a problem with this limit if I’m careful.

It uses the Embedded PL/SQL Gateway (EPG) [docs] for its web server. Nothing to report there, it worked fine right out of the box.

Next step is to upgrade to APEX 3.0 which means I lose the cute APEX database administration interface, but I have to do this because the apps I want to migrate are currently on 3.0. Not to worry, there’s a good guide for administrating Oracle XE and APEX 3.0.1 here. After upgrading I disable the SYSTEM login for APEX, and use the ADMIN account like I’m accustomed to.

After a bit of tyre-kicking I’m off to bed.

[Next]


Dirt Cheap Oracle step #2 of 6: Install Linux

[Previous]

Ok, I’ve got all the hardware I need (took a few minutes with a vacuum cleaner to carefully suck out as much of the dust that was caked in there), and hooked it all up. The computer starts up ok, but it needs an OS – the hard disk was wiped clean of all government secrets, as expected.

I downloaded Oracle Enterprise Linux release 5 from OTN [link]. I choose the x86 32 bit version, Release 5 Media Pack. It comes as five ISO images (ignoring the four Source ISOs), which I burned onto five CDs (just happened to have some blanks lying around).

Pop the first CD into the tray and hope nothing’s broken. It works! It’s alive!

Oracle Enterprise Linux welcomes me into its world and I recklessly accept all the defaults. It’s successfully detected all the hardware, and after the last CD has been inserted the transformation from boring ex-government PC into a shiny clean Linux box is complete.

A bit of a brush up on linux commands (Guide to Linux File Command Mastery), and now it’s play time!

[Next]


Dirt Cheap Oracle step #1 of 6: The Machine

[Previous]

My main computer, used by my wife as well, runs XP and I want to remove Oracle from it because it takes too long to boot up, and Oracle can suffer when I run CPU-intensive applications on it.

In spite of my asking very nicely, none of my friends, family, in-laws, or colleagues had any old computers lying around they didn’t need, so I’ve done a bit of shopping around. An ad in the local paper pointed me to McLernon’s Supply & Demand, an auction house which sells heaps of old office furniture and computer equipment. They had just stacks of secondhand computers, mostly from DOLA (Department of Land Administration) (who, coincidentally, I once worked for), and I picked up a good PC for $99. It had 512MB of RAM so I also got a second half-gig for $50.

This is the hardware manifest:

  • 1 x Intel P4 2.0GHz, with 512MB RAM, 40GB HD, onboard video & network: $99.00
  • 1 x 512MB RAM: $50.00
  • Network cable: free (was in my box of computer junk, left over following an upgrade of my router)
  • DSL-G604T 4-port+wireless ADSL Router: $86.00 (secondhand via eBay)
  • Labour: free (me)

It’ll use my existing broadband to access the Internet. I’m on a $49.95/month plan from iiNet.

TOTAL COST: $235 upfront, plus the monthly broadband cost.

(not mentioned: a KVM switch, dirt cheap for $35, just for convenience while setting it up)

From here on in, there’ll be no further outlays; free software only!

[Next]


Dirt Cheap Oracle

Seeing how Oracle XE and Linux is free, I want to see for just a minimum of outlay I could get a working database and web server running. I am also keen to give Linux a try; I’ve been a Windows user ever since my dad upgraded his computer from MS-DOS 3.3 to Windows 3.1, and while I’ve always worked for companies that used Unix I’ve never really had to learn much about it.

My goal is to get Linux up and running on the cheapest hardware I could find (free, if possible), and to run Oracle XE, the free version of the Oracle database, which includes Application Express. On the same machine I want to set up a web server to serve static web pages and streaming media.

My plan is as follows:

  1. Obtain a barebones machine to be the server. Must be free, or dirt cheap.
  2. Install Oracle Enterprise Linux on it.
  3. Install Oracle XE 10g R2.
  4. Upgrade APEX to version 3.0.
  5. Set up the router for external access.
  6. Migrate all Apex applications from the Win XP box.

In the next few posts I’ll describe how I went on all six points.

[Next]


Which APEX am I logging into?

I’ve got two computers at home both running APEX, and when I log into APEX the page looks exactly the same – except for the IP address in the URL, which is just slightly different. If you’re forgetful like me it’s easy to confuse which machine I’m logged in to. Here’s a tip:

Login to apex_admin on each machine in turn and click Manage Service, then Messages. Then select a Custom Message for the Login Message and/or a System Message, and enter the name of the machine, or some other message that will remind you which machine it is. The message will be shown whenever you see the login screen, and when you login to APEX.

If you find the text isn’t visible enough, you can insert HTML into the message, e.g. <B>Development</B>.


Un-riching Rich Text Format

Let’s just call it Legacy because I’m not going to say what the source is. It is a single-user desktop application that after a little investigation (i.e. searching the online forum for the app) was found to be storing its data in tables readable by MS Access. I wanted to get at this data, analyze it, maybe do some smart things to it, and then present it via Apex.

Step 1: Get the data into Oracle.

Simple matter of exporting from MS Access via ODBC. At least, it was simple once I replaced the Oracle ODBC drivers with the latest download from OTN. Before that I was getting a number of annoying TNS errors.

Step 2: Transform the data.

Most of the tables are easy-to-understand normalized relational tables. One of them, however, has a column that came through as a CLOB containing strange values like this:

{\rtf1\ansi\deff0\deftab254
{\fonttbl{\f0\fnil\fcharset0 Arial;}
{\f1\fnil\fcharset0 Verdana;}}{\colortbl\red0\green0\blue0;\red255\green0\blue0;\red0\green128\blue0;\red0\green0\blue255;\red255\green255\blue0;\red255\green0\blue255;\red128\green0\blue128;\red128\green0\blue0;\red0\green255\blue0;\red0\green255\blue255;\red0\green128\blue128;\red0\green0\blue128;\red255\green255\blue255;\red192\green192\blue192;\red128\green128\blue128;\red255\green255\blue255;}
\paperw12240\paperh15840\margl1880\margr1880\margt1440\margb1440
{\*\pnseclvl1\pnucrm\pnstart1\pnhang\pnindent720
{\pntxtb}{\pntxta{.}}}
...

Now based on my knowledge of the application I knew that this column was used to store small pieces of text (typically 8 to 30 short lines), with some amount of formatting (e.g. fonts, alignment, etc.). Again the online forum came in useful in that a side comment from one of the developers (regarding a small bug undocumented anti-feature) revealed that they stored the formatted text as RTF – Rich Text Format. Should have known from the opening 6 bytes in the data.

Somewhere in these oceans of rtf codes were swimming the plain text I craved. So Googled RTF, skimmed this old RTF specification, and ended up with this admittedly poorly-performing PL/SQL, which for the 651 rows in this table, each with an RTF of average 3KB, works just well enough for my purposes. As it turned out the only RTF codes I was interested in were \fcharset and \*, both of which I used to ignore bits of text I didn’t want in my output. Oh and \par, which denotes the end of a paragraph. I can run this script once a month on the freshly exported data and apply the full weight of Oracle’s analytic capabilities on it.

The code below exemplifies the use of a pipelined function. This is not a good idea, by the way, if you want to use it in regular queries, e.g. a view. In my case, however, I only wanted to call this from within PL/SQL, and then only once a month. Because of the way parameters work with functions like this, I had to call it with dynamic SQL (execute immediate).

That was kind of fun, but I’d rather not have to deal with RTF ever again, thank you.

create or replace package myutil_rtf is
  type t_v4000_table is table of varchar2(4000);
  function extract_text (p_recid in number)
  return t_v4000_table pipelined deterministic;
end;
/

create or replace package body myutil_rtf is
  function extract_text (p_recid in number)
    return t_v4000_table pipelined deterministic is

    l_ch varchar2(1);
    l_ctrl varchar2(4000);
    l_line varchar2(4000);
    l_rtf clob;

    --don't output any text between
    --\fcharset and closing }
    l_fcharset boolean := false;

    --increments for each enclosed pair of { }
    --within a discard section
    l_discard number;

  begin

    select rtf_clob into l_rtf
    from rtf_table where recid = p_recid;

    for i in 1..dbms_lob.getlength(l_rtf) loop

      l_ch := substr(l_rtf,i,1);

      if l_ch = '}' then

        if l_fcharset then
          --closing } found; re-enable output
          l_fcharset := false;
          l_line := null;
        end if;

        if l_discard > 0 then
          l_discard := l_discard - 1;
          if l_discard = 0 then
            l_discard := null;
          end if;
        end if;

      elsif l_ch = '{' then

        if l_discard is not null then
          l_discard := l_discard + 1;
        end if;

      elsif l_ch = '\' then

        --controls start with a backslash
        l_ctrl := '\';

      elsif l_ctrl is not null then

        --controls are always ended by some
        --non-alphanumeric character
        if instr('abcdefghijklmnopqrstuvwxyz'
        || '0123456789',lower(l_ch)) > 0 then
          l_ctrl := l_ctrl || lower(l_ch);
        else
          if l_ctrl = '\par' then
            pipe row (l_line);
            l_line := null;
          elsif substr(l_ctrl,1,9) = '\fcharset' then
            l_fcharset := true;
          elsif l_ctrl || l_ch = '\*' then
            --{\* ... } means you can ignore
            --anything between the { }
            if l_discard is null then
              l_discard := 1;
            end if;
          end if;
          l_ctrl := null;
        end if;

      elsif l_ch not in (chr(10), chr(13), '{')
        and not l_fcharset and l_discard is null then

        l_line := l_line || l_ch;

      end if;

    end loop;

    if l_line is not null and not l_fcharset then
      pipe row (l_line);
    end if;

    return;
  end extract_text;
end myutil_rtf;
/

To extract the text from the table with recid=1:

select column_value line_of_text
      ,rownum line_number
from table(myutil_rtf.extract_text(1));

Do you know the difference between USER_TAB_COLS & USER_TAB_COLUMNS?

I just noticed the difference – I was wondering why I couldn’t see the virtual columns (i.e. columns that support function-based indexes) on a table, and it was because I was querying USER_TAB_COLUMNS. Once I used USER_TAB_COLS instead there they were.

The difference is noted in the documentation (Oracle 10g Release 2 (10.2)) for USER_TAB_COLS and ALL_TAB_COLS but not for USER_TAB_COLUMNS or ALL_TAB_COLUMNS, unfortunately.


Replace the default XDB welcome page

I was annoyed that to get to my main PL/SQL page I have to type in a full URL like this:

http://host:7777/mydad/home

(e.g. “mydad” could be “apex” for Oracle Apex)

If I was using Apache HTTP Server I’d do something this article suggests. But I’m using Oracle’s Embedded PL/SQL Gateway.

A. I got rid of the “:7777” by changing the HTTP port to 80, e.g.:

SQL> exec dbms_xdb.sethttpport(80);

Now, I can get to it without the port number:

http://host/mydad/home

B. Now I want to remove the need to remember to type “home”. To do this, I just tell the DAD what the default page is:

SQL> exec dbms_epg.set_dad_attribute('MYDAD','default-page','home');

Now, the url is a bit simpler:

http://host/mydad

The URL is now rewritten automatically to point to “mydad/home”.

C. Finally, I want to remove the need to specify the DAD. To do this is a little more complicated. I’ll create an XDB resource that will override the default XDB navigator that comes up.

    1. Log into Enterprise Manager

 

    1. Open the “Administration” tab and select “Resources” under “XML Database”

 

    1. Click “Create” and set the fields as follows:
      Owner = SYS
      Name = index.html
      Location = /
      Type = XML Database Resource File
      Source = Specify the file contents
      Contents =
      <html><head><meta http-equiv="REFRESH" content="0; URL=http://host/mydad"></head><body><a href="http://host/mydad">Home</a></body></html>

 

  1. Click “Ok”

(you’ll need to change “host” and “mydad” to appropriate values in the sample Contents above)

Now, the url is simply:

http://host

This causes it to load the index.html page from the XML database, which redirects to the DAD, the default page for which is “home”.


APEX 3.0 via Embedded PL/SQL Gateway

I managed to get APEX 3.0 working on Oracle 10.2.0.1 using the embedded PL/SQL gateway (i.e. without installing Apache), contrary to the advice given here. I used apex_epg_config instead of apex_epg_config11. For it to work, however, I had to remove some security on XDB so it’s not a recommended option. This involved modifying my xdbconfig.xml to allow anonymous access to XDB using this:

DECLARE
  configxml SYS.XMLType;
BEGIN
  SELECT INSERTCHILDXML(xdburitype('/xdbconfig.xml').getXML(),
         '/xdbconfig/sysconfig/protocolconfig/httpconfig',
         'allow-repository-anonymous-access',
         XMLType('true'),
         'xmlns=&quot;http://xmlns.oracle.com/xdb/xdbconfig.xsd&quot;')
  INTO configxml FROM DUAL;
  DBMS_XDB.cfg_update(configxml);
END;
/

Now for some general tips:

Tip #1. List Your DAD Authorizations

As described so well in the excellent article DBMS_EPG – The Embedded PL/SQL Gateway in Oracle 10g Database Release 2, you can authorize and deauthorize a DAD using the DBMS_EPG package. It should be noted that dropping a DAD will not remove the authorizations as well, so you need to deauthorize it as well. To find out what authorizations have been made on your database, you can query the DBA_EPG_DAD_AUTHORIZATION view, e.g.:

SQL&gt; select * from dba_epg_dad_authorization;

DAD_NAME USERNAME
-------- ---------
APEX     ANONYMOUS

1 row selected.

There’s also a USER_EPG_DAD_AUTHORIZATION view which shows the DAD_NAME for authorized DADs for the current user.


Tip #2. List Your DADS

You can find out what DADs have been set up on your instance using the DBMS_EPG package. There is a bug, however, that causes dbms_epg.get_all_dad_attributes to return nothing, so this script gets each attribute separately with dbms_epg.get_dad_attribute. The script lists all the DADS, their mappings and their attributes.

set serveroutput on

PROMPT list all dads
declare
  blank       dbms_epg.varchar2_table;
  dad_names   dbms_epg.varchar2_table;
  paths       dbms_epg.varchar2_table;
  procedure show_dad_attribute(dad in varchar2, attr in varchar2) is
    val varchar2(4000);
  begin
    val := dbms_epg.get_dad_attribute(dad, attr);
    if val is not null then
      dbms_output.put_line('... ' || attr || '=' || val);
    end if;
  end;
begin
  dbms_epg.get_dad_list(dad_names);
  if dad_names.count &gt; 0 then
    for i in dad_names.first..dad_names.last loop
      dbms_output.put_line(dad_names(i));
      paths := blank;
      dbms_epg.get_all_dad_mappings(dad_names(i), paths);
      if paths.count &gt; 0 then
        for j in paths.first..paths.last loop
          dbms_output.put_line('... path=' || paths(j));
        end loop;
      else
        dbms_output.put_line('... No mappings found');
      end if;
      show_dad_attribute(dad_names(i),'after-procedure');
      show_dad_attribute(dad_names(i),'always-describe-procedure');
      show_dad_attribute(dad_names(i),'authentication-mode');
      show_dad_attribute(dad_names(i),'before-procedure');
      show_dad_attribute(dad_names(i),'bind-bucket-lengths');
      show_dad_attribute(dad_names(i),'bind-bucket-widths');
      show_dad_attribute(dad_names(i),'cgi-environment-list');
      show_dad_attribute(dad_names(i),'compatibility-mode');
      show_dad_attribute(dad_names(i),'database-username');
      show_dad_attribute(dad_names(i),'default-page');
      show_dad_attribute(dad_names(i),'document-path');
      show_dad_attribute(dad_names(i),'document-procedure');
      show_dad_attribute(dad_names(i),'document-table-name');
      show_dad_attribute(dad_names(i),'error-style');
      show_dad_attribute(dad_names(i),'exclusion-list');
      show_dad_attribute(dad_names(i),'fetch-buffer-size');
      show_dad_attribute(dad_names(i),'input-filter-enable');
      show_dad_attribute(dad_names(i),'info-logging');
      show_dad_attribute(dad_names(i),'owa-debug-enable');
      show_dad_attribute(dad_names(i),'max-requests-per-session');
      show_dad_attribute(dad_names(i),'nls-language');
      show_dad_attribute(dad_names(i),'path-alias');
      show_dad_attribute(dad_names(i),'path-alias-procedure');
      show_dad_attribute(dad_names(i),'request-validation-function');
      show_dad_attribute(dad_names(i),'session-cookie-name');
      show_dad_attribute(dad_names(i),'session-state-management');
      show_dad_attribute(dad_names(i),'transfer-mode');
      show_dad_attribute(dad_names(i),'upload-as-long-raw');
    end loop;
  else
    dbms_output.put_line('No dads found');
  end if;
end;
/

When the above script is run, you’ll get something like this:

APEX
... path=/apex/*
... authentication-mode=Basic
... database-username=ANONYMOUS
... default-page=apex
... document-path=docs
... document-procedure=wwv_flow_file_mgr.process_download
... document-table-name=wwv_flow_file_objects$
... nls-language=american_america.al32utf8
... request-validation-function=wwv_flow_epg_include_modules.authorize

ZFS/Oracle

Zettabyte File System (APC)

Pooled storage, transactional control, snapshots… sounds awfully familiar to me. Oh yeah, features that Oracle have provided in the database for years. 🙂

(I know, very different domain and probably very different implementation (database vs. file system) but some of the concepts are similar.)