A Land Far Away

1984L Project Specifications

Background:
In order to prepare the following proposal, I read all the available threads concerning the proposed use of a logging system in ALFA2. I have tried to incorporate as many suggestions and ideas concerning this as possible. When opinions differed, I have necessarily made some decisions in order to forward a complete project proposal. Nonetheless, if members of the community feel strongly that elements of this proposal will not meet their needs, please inform me ASAP so we can quickly resolve a satisfactory plan.


Structural Design:

I. In-game hooks to capture data
a. Description: A process, consistent with the existing NWN2 scripting environment, to extract data from within the game environment.
b. Responsibilities
i. It is the responsibility of the 1984L team to provide a standardized format for data output.
ii. It is the responsibility of individual team leaders to implement that format into their builds and update the necessary resources.

II. Centralization
a. Presumably, the different N2 servers will be running on different machines at different locations. Extracted information will come off the servers locally and be passed to a central location for data parsing and storage.
b. Responsibilities
i. It is the responsibility of the 1984L team to provide the tool, installation and maintenance instructions for each team to run the tool with their server.
ii. It is the responsibility of individual team leaders to ensure the tool is implemented with their server.

III. mySQL driven database
a. A database structure containing the log data for the ALFA2 world-base.
b. Responsibilities
i. It is the responsibility of the 1984L team to define and maintain this resource.

IV. PHP driven web-based access
a. A web-based interaction that allows direct connectivity to the central database.
b. Responsibilities
i. It is the responsibility of the 1984L team to maintain this resource and to interact with the community to provide services through this resource.

V. In-game hooks to retrieve data
a. A process, consistent with the existing NWN2 scripting environment, to retrieve data from the central database.
b. Responsibilities
i. It is the responsibility of the 1984L team to provide a standardized format.
ii. It is the responsibility of individual team leaders to implement that format into their builds and update the necessary resources.


Data Structure: (Draft)

Goal – develop a core data structure and data handling mechanism that allows a minimum data logging system to be deployed and easily scaled up to contain additional logging information. The key things to define here are the differences in when and how we extract various data types from server. For example, if we want to capture things like player transitions in the game, that data is very dynamic and has to be captured during actual play compared with getting a dump from stored items or parsing the character vault.

I. Network information
a. IPs
b. Login
c. Activity
II. Player Character Information
a. Stats
b. equipment
c. gold
III. Persistent Storage
a. chests
b. bank
c. houses
IV. Gameplay
a. Combat
b. Transitions
c. Zones
V. Transients
a. Trade transactions
b. Drops
c. pick-ups


Proposed 1984L Services:

I. Security Access and Backup
a. Access to the DB will be limited to 1984L and designated ALFA2 administration.
b. Routine backup and DB cleanup will be performed

II. Automated alarms
a. Certain conditionals will be set to generate automatic server-wide DM alarms

III. Standard reporting
a. A standardized reporting format will be available to each server team leader.
b. The report will be available on the web and have certain services to facilitate information review.

IV. Petition for information system
a. A means for requesting log information will be provided to all DM’s to support investigations of player abuse.


Development Plan:

I. Alpha
a. Goals
i. Define the smallest test data set consistent with expansion.
ii. Select a single server team to test 1984L systems.
iii. Develop prototype resources and basic implementation for In-game hooks to capture data, Centralization/Parsing, and mySQL database.

II. Beta
a. Goals
i. Work with other server team leads to get minimum 1984L embedded into their builds.
ii. Develop and test complete network resources.

III. Core System (Live)
a. Goals
i. Deploy beta 1984L system to live servers
ii. Enable basic web-based DB access with Petition for Information Service.
iii. Develop and test In-game hooks to retrieve data

IV. Expansion Plan (Post-Live)
a. Goals
i. Develop and test advanced reporting services
ii. Develop and test automated alarm service
iii. Deploy services as they come online


So, there it is… my first pass at this. In summary, my first goal would be to categorize all possible log data into four categories:

Test
The absolute smallest set of log data that represents the different ways we will extract data from the servers. This should be way fewer elements than what people think is the least amount of log data necessary to do basic play-server surveillance. The most important element of this data set is its technical diversity (i.e. when/how is this data extracted from the server?). If we are going to extract login attempts, that is different than extracting data that streams out live from the game (such as items PC’s trade, drop or place in containers), and different than doing a data dump on logout or parsing the character vault. We need to come up with a single prototype for each of these data types and use them to test the overall build.

Core
This is the minimum data set that people think is necessary for the servers to go ‘live’. This should not encompass all exploit or data-mining possibilities, but instead impart some security for the obvious things while providing server leads with important aggregate statistics of their game.

Expansion
All data types that can be easily expanded into the existing data structure and are considered desirable and useful to the community. We will prioritize this list by desire and ease of incorporation.

Development
Data types that pose technical problems to the existing structural design and require significant R&D to get them working.


Ok, so that is it. *Braces himself for criticism*

There's nothing to criticize. I think it's a great high level document. I'm assuming you're willing to drive this effort through to implementation, in which case you should start by first understanding and determining how best to record the requisite game data.

I spoke with Hialmar and he's available if you need to consult with him. He'll be able to discuss infrastructure issues with you (like setting up and accessing a central database or setting up suitable web services to receive log transmissions and generate real time reports). On the ACR side of things, the logging will likely occur on specific events. Every event in the game will have an ALFA event script attached to it (what we call an ALFA Configuration File or ACF) so triggering the log functions on these events should be pretty straightforward. Just contact me (or Ronan) if you ever have any questions about the ACR framework.

Thanks for jumping in and taking this bull by its proverbial horns.

No problem. I will take over as 1984L lead. Thanks for the support. I've already put a msg into Hialmar; he will obviously be a great avenue of direction and advice.

On the topic of getting game data to the database, I spent most of last evening going over the available possibilities with a network DB professional. He will become a defacto 1984L team member even though he doesn't play NWN. The most obvious and straightforward plan is to implement NWNX4 SQL output directly to a centralized mySQL DB that I host on the net.

I think there are two concerns about using NWNX4: 1) it is a 3rd party application that will need updating with alterations to the game engine; and, 2) the use of internally driven SQL writes to a remote server may use more game server resources than a log write and transfer.

On Concern #1: No matter how this nut is cracked, it will require a 3rd party software layer to get game data into a usable DB. If we do a log/transmit/parse scheme, we will have to write and support our own internal code through the lifetime of the ALFA2 project. On the other hand, in my experience, NWNX has been one of the best supported OEM NWN products since the original game came out. The NWNX lead, Papillion, is fantastically responsive and very helpful when it comes to problems. He is very quick to issue patches in response to game server changes; and, in theory, Obsidian is even being considerate to NWNX as they update (something Bioware never even paid lip service to). I have the newest version of NWNX4 and it works right now.

On Concern #2: All logging schemes will cost game server clock cycles. I am in the process of evaluating the relative game server cost for direct SQL writes vs. a logging/transmitting scheme. I'm trying to set up a test in my home network to try it out. If NWNX direct SQL writes costs significantly more server cycles and does not scale well with increasing data streams (more log data as we expand our logging capabilities), it would be reason to adopt the risk of developing and supporting our own internal logging/transmitting/parsing scheme. I'll let you know what I find out.

Thoughts and comments are welcome.

The NWNx proposition is of course a good one. Our biggest hangup with a centralized database has been single point of failure. If it goes out, what happens? Can we still play the game? The loss of logging functionality isn't critical to game play, but I do think it would be prudent to use the alternate log/transmit/parse scheme as a failover method when/if the database becomes unavailable. The logs can be imported back into the database when it comes back up so we never lose data just b/c the db was offline.

Furthermore, an SQL database can really allow us to do some fun things in game and out, so I'm very much in favor of figuring out a way to incorporate one reliably, not just for logging purposes.

It's great to see the ball moving forward on this.

Good show Daeftan.

Check this thread out...maybe Hilamer's already linked you up to it. It's a conversation we had a few months back.

http://www.alandfaraway.org/phpbbforum/ ... hlight=log

Yes, thank you indio. I read it in detail before preparing my proposal. While I couldn't accomodate all the issues, I hope I covered the majority of dev/DM/player concerns. Please let me know if there is something glaring I am leaving out.

As far as the mySQL server going down... I will put the server up on a professional host that is managed 24/7. Nonetheless, that does not mean it will guarenteed be up 100% of the time, nor that the TCP/IP connection will be 100% available.

As far as the game server goes, my current experience with the NWNX4 system is that the game server will continue to run without problems if the DB service goes down. In most cases it will reconnect automatically when the DB comes available again and resume data transfer. Data will be lost in this event if no additional buffering process is put in place; however, since we are not currently planning on using the 1984L database for critical information needed by the live server, I'm not sure how much effort we should put into data redundancy (something I broke my teeth on at GHL since we used the SQL DB to store player inventories in banks and houses).

I will have a mini verison of this setup in my home this weekend and will test it for performance and reliability. I agree that a real SQL DB will be awesome to have. It will allow some really great data mining on our servers. I think it will help not only in the 1984 capacity, but also help us understand what players are doing and how we can develop to help them have more fun.

On a side, love that avatar.

Update on 1984 Logging System:

I have installed a test system at my house. It is three computers: one runs the server, the other a mySQL DB, and a third is for client-logon and testing. I am running NWNX4 v1.07 as a system resource. All machines are running Windows XP. Here are some preliminary observations:

1) The database is extremely fast compared with the NWN2 internal db system. This is true whether mySQL is running on the same machine as the server, or it is running on a seperate machine and using UDP/TCPIP connection sockets.

2) The system is very stable. I have purposefully crashed the mySQL server in a number of aggressive ways during active operation (i.e. unplug the internet cable, unplug the computer, run processor denials, etc.). I have yet to see an example of NWNX4 crashing the game server. Obviously, if the game is requesting information from a crashed DB, it will not receive it. Also, if the SQL db goes down in the middle of write operations, it will loose data (I have no read/write buffering in place).

3) Scripts to log data within the game are very easy to implement. The Set/GetPersistentVariable functions work great as does direct command line SQL commands issued from various scripts.

What I'm doing now:

I am coding up some test scripts to run high volume traffic between the server and DB to test its capacity (lots of reads and writes in for loops with time stamps).

I am doing more testing on running the mySQL db directly on the server platform. With the game server set to "real-time" priority on a dual-core machine, I think you can run the DB in the back ground with almost no cost to performance. Doesn't really help us centralize 1984 data from the entire ALFA2 world, but it is interesting for other potential applications of an external mySQL db.

I am working on a mini-module that allows DM's to test the game server to DB connections while in game. Right now it is just a small interior room with a couple of NPCs that report connection data via conversations. Just something simple right now to play with.

Thanks for the update. Appreciate your thoroughness Daeftan.

Since you're testing out the database, have you given any thought to schemas? Something along these lines perhaps:

Players table
id | gsid | cd_key | valid | banned

Characters table
id | player_id | name | race | class | alignment | date_created | valid

Logs table
id | character_id | event | value | date_created

The logs table can contain data on just about anything, from logins to level ups.

*Stickied and Updated*

How are things going Daeftan?

*bump*

Any progress or updates, Daeftan?

He's AWOL. Last sighting was two weeks, stating he was back due to an RL situation. Then *poof* gone again. So I'm guessing RL is keeping him away indefinitely.

Rats. That's bad news.

Thanks for the heads up though BW.