How to get more out of ODBC databases and Akiva's WebBoard with Rog's OSQL batch extractor (version 1.0)

The OSQL batch interface lets you extract data from ODBC compatible DBMSs in "commas-n'-quotes" format (or fixed-length format) via batch processes.

It also lets you dump all the base table layouts (i.e. excluding views) in a database to a plain text file--with just one DOS command.  (If you wish, you can also dump a single base table layout).

And there's a special interface that allows you to dump out all the data in all tables in a DB into plain ASC text ("commas-n'-quotes" format) with just one command.  If you use WinZip to compress this data, it can also be very useful for "partial restores," i.e. cases in which you just need to get a few records back, but you don't want to restore the entire DB (e.g., because it's on a production server).  You can also dump a single table's data out in "commas-n'-quotes" with just one command.

WebBoard users: If you're a virtual board manager, or anyone else who doesn't have server access, this system can make it a lot easier to get data from your sysadmin, since s/he can give it to you with a minimum of effort.  If you can program in a standard programming language, you may also benefit from an executable that converts commas-n'-quotes format to fixed length format.  This system requires no special external software to run: it relies only upon OSQL.Exe, which is standard with MS's ODBC interface.

WebBoard users: the system comes with a text file that lists all WebBoard base table layouts (there are no views in WebBoard).

This system requires no special external software like Enterprise Manager to run: it relies only upon OSQL.Exe, which is standard with MS's ODBC interface.

This code is freeware.  You use it at your own risk.  No warrany of fitness for use or any other form of guarantee inheres.

You may use it and/or the source code in whole or in part for any otherwise legal purpose that you deem fit; however you may not claim legal ownership of same for legal purposes against anyone who's obtained it from an "independent source" (such as this web page).


1:  Install

2:  Example

3:  Getting accurate table layouts

4:  "Clean" and accurate layouts for all base tables in a DB

5:  Getting all the data in a DB, and/or all table layouts

6:  Getting just one table layout, or just one table's data

7:  Miscellaneous

8:  An important limitation

9:  Notes for WebBoard users

10:  More about DOS batch files and my utilities

11:  [A]sc [D]ata [B]atcher

12:  Bug reports, comments, change requests, and how to hire me

13:  Other freeware products

14:  Version history information

15:  Acknowledgement


1:  Install

OSQL.Zip contains five executables: DOSDate.Exe, DBCvt.Exe, OSQLRf1.Exe, OSQLRf2.Exe, OSQLRf3.Exe, and Repl.Exe, well as the source C code for each.  This documentation file (also included) is OSQL.Htm, and there are two batch files: OSQLTab.Bat, and OSQLDmp.Bat.

For WebBoard users, an additional file called OSQLWBDB.Txt shows the layouts for the most important tables in the WebBoard database.  (This is for WebBoard 4: OSQLWBD5.Txt is for WebBoard 5, OSQLWBD6.Txt is for WebBoard 6, and OSQLWBDB7.Txt is for WebBoard 7.

If you don't have server access, then only OSQLRf1.Exe needs to be run by the person who can get to the server.

Return to table of contents

2:  Example

Here's a sample DOS batch file which runs all three programs and shows how they can be used together.  DOS batch files end with ".bat" and can be invoked directly from the desktop with a shortcut.  You need to create them in Windows Notepad, or some other mechanism for building "plain DOS text" files.


OSQL -U sa -P -O -i Query.Dat -d WebBoard -s ¯ | OSQLRf1 -S175>Raw.Dat
OSQLRf2 <Raw.Dat >CQ.Dat -S175 -FLayout.Txt -T256
DBCvt CQ.Dat Fixed.Dat -NOQUOTES


This execution example shows three steps.

In step #1, OSQL is used to run the file Query.Dat against the database called "WebBoard."  The DBO (DB owner) name for this is "sa", which is the default MSDE DBO, and the password is nothing (blank).  Your sysadmin should know the DBO name and password for your data.

A typical query might be something like this:


select * from users

(If you don't know SQL, you can enter "+SQL +tutorial" in Yahoo's search engine to get a fairly long list of free tutorials.)

The -s parm to OSQL is followed by a space and a single character that will be used to separate the columns.  In this example I've chosen decimal 175 (which doesn't come out the same way in HTML as it does in plain text).  That character can't appear in the data.  You may also use a single character (other than a digit).  If you don't have a text editor that will let you type control characters, use DOS EDIT at the DOS command line.  To get that character, hold down the alt key on your keyboard and type "175" on the numeric keypad (be sure that scroll lock is off.)

The first step is all that needs to be run by a sysadmin who has server access: the output file (Raw.Dat) is relatively compressed in the sense that it doesn't have the unnecessary spaces, etc. that OSQL outputs.  It should be even smaller after being zipped.  Note that the column separator character is being fed to OSQLRf1 in the -S parm  (the "S" can also be lowercase).


In the second step, OSQLRf2 converts Raw.Dat into commas-n'-quotes format.  (That's the file CQ.Dat).  OSQLRf2 also takes the -S parm, and it has the same meaning as for OSQLRf1.

This format is ready to be loaded into many different kinds of interfaces.  The -F parm specifies a layout file, which shows the names of the fields and their lengths (beware that some fields may show up as longer than they really are).

The -T parm allows you to truncate fields automatically: in this case the fields are truncated at 256 bytes, which just happens to be the default.


The last step converts CQ.Dat (commas-n'-quotes format) into fixed-length format.  Both files are plain text ASC flat files.  Programmers might wish to use this "generic" format so that they can use any standard programming language to process the results.  The -NOQUOTES argument specifies that the fields are to be aligned next to each other, with no intervening commas-n'-quotes.

Return to table of contents

3:  Getting accurate table layouts

OSQLRf1 isn't very "smart" about how it computes the lengths of fields: it simply looks at the column headers that OSQL supplies.

For that reason, if the name of the field is longer than the field value, the former will be used in the layout file produced by OSQLRf2.


If you need more precise layouts, you can use the built-in stored procedure sp_help.

Here's an example of how to do this:


osql -U sa -P -Q "sp_help users" -d WebBoard -s # | OSQLRF1 -Q >Out.Txt

That example will dump out the users table into Out.Txt.

The format isn't very pretty, but at least you'll know the exact field types and lengths.

You can also use sp_help on any triggers, constraints, etc. which you'll see listed.

As in the previous examples, I assume that your DB owner is "sa" (follows the -U command-line argument), and that your password (follows the -P command-line argument) is the default of blank.


If you want better formatting, you can use the batch file OSQLTab.Bat

You simply execute it from the command line with the name of the database and the name of the table, e.g.:


OSQLTab WebBoard User

This example will print out the user table in the DB called WebBoard.

Output will be appended to the file OSQLTab.Txt, which will be created as new if it doesn't exist.

Two additional programs, Repl.Exe and OSQLRf3.Exe, are invoked in this batch file.

Here's what the batch file OSQLTab.Bat does in order to produce a table layout, in case you want to use this strategy in your own processes:


OSql -U %DBO% -P %PWD% -Q "sp_help %2" -d %DBNAME% -s # | OSQLRF1 -q >T1.Dat
OSQLRF3 -q < T1.Dat > T2.Dat
DBCvt T2.Dat LayOut.Txt -q -noquotes -varnoflds

After executing this example, LayOut.Txt will contain a fairly readable layout.  The DOS batch file variable %DBNAME% is the database name, the variable %2 is the table name, the variable %DBO% is the DB owner, and the variable %PWD% is the password.

The -q parm to OSQLRf1 suppresses the standard program "help" printout.

Note that this works for base tables only: not views.


Return to table of contents

4:  "Clean" and accurate layouts for all base tables in a DB

If you're like me, you might want to view all of the base table layouts for a DB, in a readable plain text format.

The batch file OSQLTab.Bat also serves this need.

You can do this by leaving the table name off the call:


OSQLTab WebBoard

OSQLTab.Txt will contain all the table structures for the database WebBoard.

If you've changed the Database Owner, or the DBO's password, you'll have to configure the batch file, by changing the variables DBO and PWD.

Use Windows Notepad (not a document editor) to change this file.  Or if you insist on using something like WORD, make sure you save it in "plain DOS text."

Return to table of contents

5:  Getting all the data in a DB, and/or all table layouts

The DOS batch file OSQLDmp.Bat will dump out both the data from all tables as well as (optionally) the table layouts from all tables.

It puts this into a subdirectory with the name YYMMDD.

The configuration information tells you how to use WinZip to zip this data up, if you want to use it for a kind of "partial" backup.  (You'll find this information at the beginning of OSQLDmp.Bat).

The advantage here is that you have all the tables' information in plain text, and you can easily restore just a few records without having to restore the entire database.

If you're a WebBoard administrator, and you're using this for backup purposes, remember that you'll also need to backup the /WebBoard/Confs folder as well (the *.htm files contain the message text: they're in subfolders of this folder).

If you're a developer, you can read the configuration section of this batch file to see how to speed up the execution by setting just one variable: in that case it won't bother dumping out the table layouts, but will instead deliver only the data.

Use Windows Notepad (not a document editor) to change this file.  Or if you insist on using something like WORD, make sure you save it in "plain DOS text."

Return to table of contents

6:  Getting just one table layout, or just one table's data

To get the layout of the Users table in the database WebBoard, simply type this from the DOS prompt:


OSQLTab WebBoard Users

Output will be written to OSQLTab.Txt.


To get the data in the Users table in the database WebBoard in commas-n'-quotes format, simply type this from the DOS prompt:


OSQLDmp WebBoard Users

Output will be written to Users.Dat.


Let me again remind you that both these batch files must be configured if you've changed the DBO (database owner) from the default of "sa" or the DBO's password from the default of blank (i.e. the null string, AKA nothing).

Use Windows Notepad (not a document editor) to change these files.  Or if you insist on using something like WORD, make sure you save them in "plain DOS text."

Return to table of contents

7:  Miscellaneous

All of the programs and batch files print running instructions if you execute them from the DOS command line.

All of the executables require DOS file names.  That means up to 8 characters plus a period and up to 3 more characters.  I suggest you limit yourself to letters, digits, underscores, and dashes.  Do not use spaces.


Leading and trailing spaces aren't preserved in the data.


You can run exactly one query (SELECT) per execution of OSQLRf1 and OSQLRf2.  If you need to extract the results of more SELECTs, you'll have to clone the example code for each one.


Your syaadmin may be (understandably) reluctant to run an executable on your server, unless s/he is absolutely convinced that it doesn't contain a virus, "Trojan horse," or some other method involving unauthorized entry.  Of course, it's possible to run OSQL by itself and reformat the output on another system.  The output file will be suitably small, if zipped.  Nevertheless, OSQL writes very large files, and if you can use the C source or this web page to persuade your sysadmin to run OSQLRf1, then you'll see significant efficiency improvements.  Note that the string "open" doesn't appear in the C source (because all the I/O is done though standard input and standard output).

My hope is that most sysadmins will appreciate the fact that they don't have to start any special software: they just click once on a DOS batch file icon and send out one file . . . this makes it easier and more convenient to set up periodic extraction processes.


Because commas-n'-quotes formats vary widely when it comes to the syntax used for double quotes in the data, OSQLRf2 will convert double quotes to single quotes by default, or use the replacement character of your choice.  You can use the -D parm for this: the syntax is the same as for the -S parm (a 1-3 digit number for decimal hex values, or a single character, other than a digit).

Return to table of contents

8:  An important limitation

For the most part, the OSQLRf1/OSQLRf2 combination will handle linefeeds and/or carriage returns in the data.

There's an unfortunate exception to this, which has to do with how Microsoft's OSQL program handles the column (field) separator character.

When the -s parm to OSQL is used to specify a field separator character, the character will not be included if the field (without the character) ends exactly in column 72.  I suspect that the field separator character option was added to OSQL after the program had already been written . . . and some programmer just ran out of time to complete the project.  (It has to be a "bug," rather than a feature, because even fields at end-of-line do have separator characters.)

OSQLRf1 uses an elaborate strategy to work around this problem.  If you code the -S parm, it will look for a line that doesn't end in the field separator character, and which is followed by a line that begins with a tab in column 1 and a non-blank in column 2 . . . when reading the column labels.

That's fine for the labels, but it won't work for the data.  So OSQLRf1 keeps track of the offset of the line(s) which require appending of an additional field separator character.  As it turns out, each record is always printed on exactly the same number of physical lines . . . if there are no carriage returns or line feeds in the data!

Hence, in the "pathological" case in which a field ends exactly in column 72 (and therefore lacks a separator character), and a preceeding field has carriage returns, OSQLRf1 will improperly split the fields!

Thus Microsoft's implementation of the field separator character has made it particularly difficult to "generically" dump out all the data in a table or in all the tables in a database.

What's needed is a stronger implementation: one that uses the stored procedure sp_help to dump the table layout.  Then the proper SQL query can be generated in order to gaurantee that the column separator character follows all-but-the-last field.

Depending on my experience with this system, future releases may incorporate such a patch.

Return to table of contents

9:  Notes for WebBoard users

For WebBoard users, an additional file called OSQLWBDB.Txt shows the layouts for all base tables in the WebBoard database.  (This is for WebBoard 4: OSQLWBD5.Txt is for WebBoard 5, and OSQLWBD6.Txt is for WebBoard 6.)

Many WebBoard applications require that you dump out all the information in the users table.

For various reasons, you may wish to limit the search to the users for just one board, and/or exclude the WebBoard administrator (in order not to disclose his or her password).

The following SQL query selects all users except user 1 (the administrator) from board 6:


select * from users as u, board_user as b
         where b.boardid = 6 and b.userid = u.userid
               and u.userid != 1

You can discover your board ID# by putting this JavaScript on any of your board's pages:


<SCRIPT LANGUAGE='JavaScript'>
alert(document.cookie)
</SCRIPT>

Look for the WB-BoardId cookie.  This cookie is set even on "no authorization" boards or "basic authorization" boards (provided that the browser will accept cookies).

If all else fails, WB's Boards table will have the board ID#.  There is no sensitive information in this table for most applications, so you can ask your sysadmin to dump it out for you.

Return to table of contents

10:  More about DOS batch files and my utilities

I have a tutorial on DOS batch files, in the documentation for ADB, my [A]sc [D]ata [B]atcher (for more on ADB, see section 11).

I also have a section that covers my DOS batch utilities.

The DOS batch file primer is at: http://www.rs-freeware.org/adb/adbdos.htm

The documentation for my DOS utilities is at: http://www.rs-freeware.org/adb/adbutl.htm

The time you spend learning how to use DOS batch files will be quite well-spent, IMO.

Even if you're a seasoned programmer, you might be rather surprised to discover how many operations you can perform more rapidly via the command line.  All standard programming languages (even the dreaded VB) contain a "shell" interface, as I'm sure you well know.  You can also "shell" using ASP or ASP+.

Return to table of contents

11:  [A]sc [D]ata [B]atcher

The layout file that OSQLRf2 generates is in the proper format for another one of my freeware products, called the [A]SC (text flat file) [D]ata [B]atcher.

The example in section 2 generates a file layout for ADB.

ADB's available at:
http://www.rs-freeware.org/adb/

You can find out more about it by browsing to that link and clicking on the FAQ page.

Return to table of contents

12:  Bug reports, comments, change requests, and how to hire me

You can e-mail me at rog@NOSPAM_rs-freeware.org if you encounter problems, or think that you've found a bug.

I can also be reached by telephone in the U.S. from about 9 AM U.S. central time to 5 PM U.S. central time at: 765-742-6705.  If you don't get an answer, you can use my numeric pager at: 765-417-0664 ... I'll try to call you back, if you're in the U.S, Canada, or Mexico.


Please supply me with as much information as you can about your server, any browser or Operating System that was involved  (9x, NT, 2K, XP?).  Keep in mind that I might actually have to have the opportunity to try what you were doing, in order to diagnose the problem.


BTW, I don't gaurantee to answer all e-mail or fix all bugs, etc.  Please remember that this is freeware, and that my time and resources are limited.

That said, I've put a lot of work into designing, coding, testing, and documenting this product, and I'm probably going to be fairly interested in any comments anyone has, fixing any bugs, and/or extending the scope to applications that strike me as being potentially valuable to a large number of users.


Naturally, if you're willing to hire me to make changes for your special-purpose application, I'm certainly willing to consider your offer.  My standard rate is $75/hr., but I may charge less if the work is to be done for a small business (less than 25 employees), or a nonprofit organization (in the latter case, I might even do it on a gratis basis  :-)  I may also consider charging you nothing if you're suggesting an improvement that I feel is of value to a large number of other users.

You can find out more about me, including references, and a list of clients/projects at: http://www.rs-freeware.org/rog.

Return to table of contents

13:  Other freeware products

WebBoard users will find lots of WebBoard freeware at: http://www.rs-freeware.org/freeware.htm.

Anyone who writes JavaScript  (regardless of whether this is written on the client or server side),  deals with SQL, or who happens to be interested in obtaining the full power available from DOS may wish to check out http://www.rs-freeware.org/free2.htm

Return to table of contents

14:  Version history information

This is version 1.0, released on August 15, 2000.

Sept. 25, '00: added the note on how to use sp_help to obtain more precise information about the table layouts.

Sept. 28, '00: added the batch file OSQLTab.Bat which dumps an individual (base) table and/or all the (base) tables in a database.

Oct 5, '00: fixed bug in OSQLRf2 that caused it to bomb out on empty lines (this affects situations in which you're dumping out just one field, and that field is the empty string), and a misleading parms display in OSQLRF1.

Oct 7, '00: added the "-O" operand to calls to OSQL.Exe in the examples.  It turns out that if the last byte of a field (other than the last field) occurs in column 73 (this could appear as column 80, if tabs are converted to the default of 8 characters), then OSQL.Exe will omit the trailing columm separator: this in turn causes OSQLRf2.Exe to become confused and to erroneously concatenate two fields.  The "-O" operand to OSQL.Exe blocks what's known as "Auto console width scaling."

Oct 7, '00: added the OSQLDmp.Bat batch file, which dumps out the data as well as the table structures in a DB, or just the data in one single table.

April 20, '01: patched OSQLRf2 to replace line feeds in the data with the string "{CR}". This makes it easier to SQL "TEXT" fields.

April 28, '01: patched OSQLRf2 to signify empty string fields by two double-quotes instead of the null string.  This struck me as being compatible with the most existing software.

April 30, '01: patched OSQLRf1 to add the -S (field separator character) . . . and to use the elaborate algorithm described in section 8 to detect the case where a field ends precisely in column 72 and OSQL omits the field separator character.

November 1, '02: Added the table layouts for WebBoard versions 5 and 6.

January 11, '05: Added the table layout for WebBoard version 7

This documentation was generated by Rog's FAQHack: a DOS/Windows-based freeware program that handles simple macro preprocessing with special support for FAQs and other structured HTML documents.

Return to table of contents

15:  Acknowledgement

I'd like to thank Dan Gumm, the Server Farm Manager at SecureWebs, Inc. in Coleville, WA, for his cooperation in testing and developing this product.

Return to table of contents