Preface
In this article I’ll describe a way to make your demos and other applications look nicer (well, at least on disk). It is intended for people who seek a powerful tool that will help them to pack all program data files into one compact datafile. We first encountered the problem in 1995 when we were releasing our first demo 90210. Here I’ll describe solutions we’ve found and provide you with a complete solution to make your releases more professional, smaller and safer in terms of reverse engineering.
Introduction
Demos use a number of various files (music modules, pictures, configuration files, etc.). Files are often organized in folders. In development phase, there is no need for a compact datafile, demo just reads files it needs from their (mostly hard-coded) locations. This is OK since nobody wants to spend any time thinking of datafiles, coders concentrate on effects and other people do their jobs. But there will come the time to release the whole thing. You find that your masterpiece uses a number of binary and text files from all over the disk. Now it is the time to use a tool that will pack all files into one that will accompany your executable to make the final release. Our solution does exactly that ... and more ... read on.
Description
Our tool called ”clib I/O datafile interceptor” (CDI) uses a simple but powerful technique that reduces time needed to release your demo to minutes. It consists of three things:
· Datafile manager (DM) that creates one datafile from all your data files
· I/O interceptor library module (IOLB) you will link with your project (sources are included) and that will intercept certain I/O calls in your program
· DOLZIO.H header file that you’ll need to include in every source module which will read a file from disk
DM is a very simple archive program. It’s simple because we just didn’t have the need to improve it any further and because I like simple programs :-). It uses simple (user adjustable) compression, handles wildcards, can compress subdirectories and also has support for list files. Most importantly, it creates a datafile that can be read by IOLB. More detailed usage description can be found in the downloadable CDI.ZIP package.
IOLB together with DOLZIO.H form the core of the package. DOLZIO.H contains #defines that map your original clib file I/O fxxx() calls to lz_fxxx() calls. IOLB contains definition of these functions. Notice: you will need NASM 0.98 assembler to re-compile LZO1X_F1.ASM in IOLB project (object file is included). And, of course, CDI fully supports long file names.
Usage
To make your demo read files from datafiles made with DM, follow these steps:
· include DOLZIO.H in every source module (.c, .cpp) that uses I/O functions (include it after STDIO.H)
· link DATAFILE.LIB with your project
· place call to open_datafile() somewhere at the beginning of your program
That’s it! Simple and easy. But beware! Not all standard I/O C functions work with files stored in a datafile. Here is the list of functions that can safely be used and work on datafile files: fopen, fclose, fread, ftell, fgets, fseek, fwrite, _stat, fcloseall. IOLB supports more than one datafile. Simply call open_datafile() multiple times to open another datafile. IOLB fopen() function will search all opened datafiles for the file you wish to open. It first searches the last opened datafile. If the file is not in this datafile, fopen() searches the last but one opened datafile .... and so on all the way to the first opened datafile. If the file is found it is uncompressed and read from a datafile. If the file is not contained in any datafile, fopen() tries to read it from disk using standard DOS fopen(). It this also fails an error is returned. In case a file is opened from datafile a fake FILE pointer is returned (ranging from 1 to n where n is the total number of all files in all datafiles).
Tips and comments
· if you use any other I/O functions (than those supported) in your program you have to rewrite your code to use only those supported ones or write your own replacement lz_xxx function
· use multiple datafiles to patch your previously released data. See sample for more details
· include datafile support in any phase of your project, even in the beginning. There is no harm if no datafile exists, your files will be simply read from disk as usually. Lately if you decide to make a datafile, you won’t have to make any changes to your project, the files will be read from the datafile instead of disk
· check return values of all I/O funcions. If an error occurs it indicates your datafile is damaged. DM generates a checksum of every 32k of your data that IOLB verifies
· do not open and close same files frequently. Instead read the whole file in memory and parse it. Closing files forces the decompressor buffer to flush thus opening the file again forces it to decompress again. Use fseek / ftell if you need to, they are far more faster than opening and closing.
· use soffset parameter in open_datafile() to produce very compact distributions (one large .EXE file for example) or to merge multiple datafiles and/or DLLs together
Conclusion
I suppose every established demo group has already developed their datafile management system. Feel free to use this source code and examples in your projects as long as you credit 7Gods. We’re open for any help making this library better and more powerful. Sources and projects are written for Microsoft Visual Studio 6.0. Porting it to other platforms should be no problem and we plan to release this library for Linux in the future. This library or its portions and modifications were used in our projects WHY and FATE.
If you have any comments to this article or release files and sources please write to sayza@7gods.org or alternatively to 7Gods@7Gods.org.
Happy coding!
written by Sayza
Copyright (C) 2000 7Gods demo group, http://7gods.org