New ACR Projects...

Scripted ALFA systems & related tech discussions (ACR)

Moderators: ALFA Administrators, Staff - Technical

User avatar
AcadiusLost
Chosen of Forumamus, God of Forums
Posts: 5061
Joined: Tue Oct 19, 2004 8:38 am
Location: Montara, CA [GMT -8]
Contact:

New ACR Projects...

Post by AcadiusLost »

Going to start cleaning this forum up and using it to organize present and future work on the core script package. Hopefully, this is also facilitate getting some more volunteers working on specific projects so we can get more of these systems into circulation on the Live servers.

Item Customization via XP-Craft: Paazin has been working on this, initially in order to allow DMs to modify the look of certain items in-game. The Customizer GUI has already been added to our GUI hak, and the major hurdle remaining is in compiling exhaustive lists of the available item appearances to code into the system. Later stages will involve implementing stations in OOC areas of existing mods, and potentially making scaled-down versions for PC use in a more IC fashion (or in an OOC fashion for new PCs/start areas).
  • Assigned to: Paazin
Portalling Improvements: ActivatePortal now works for "seamless" portaling between modules. Should be pretty straightforward to adapt the current server portal scripts to use the new function, alleviating the need to log out of one server and into the next. This may need some coordination with development work on Vaultster that Zebranky and Hialmar are working on presently, but might well function as an independent system.
  • NWN2 scripting experience: Moderate
  • ACR/NWNx4/MySQL host setup required
  • Hosting/infrastructure needs: Moderate (2 or more machines needed to test)
  • Priority level: Low (convenience factor)
Currently unassigned.

ACR Optimization: Currently a number of ACR systems involve frequent or redundant queries to the central MySQL database. Hialmar has some ideas on how to make things more efficient from the infra side of things, but there is cleanup that can be done inside our ACR code, especially during the login process (and in terms of combat logging, though I'll split that into it's own project.
  • NWN2 scripting experience: Moderate
  • ACR/NWNx4/MySQL host setup required
  • Priority level: Medium (scaling up as we get more Live servers)
Currently unassigned.

Combat data logging optimization: Presently during each OnDeath() event of an NPC or creature killed by PCs or their associates, a flurry of records is pushed across to the remote SQL database documenting how much XP was earned by each PC in the party, who the party leader was, and what was killed. In a large-party situation with multiple opponents, this can lead to significant lagspikes at critical moments. It may be possible to store combat records as local variables on a dynamically created waypoint in the area until combat is finished, then push them out onto the database after confirming no one onserver is currently in combat. As an aside, such a system could also facilitate later development of a system to award XP based on Encounter Level, rather than the individual CR of each monster. Also possible there may be improvements that can be made to how the log records are sent through the MySQL plugin such that they have less overhead for the nwn2server (or can be sent collectively as a transaction instead of individually as separate DB accesses).
  • NWN2 scripting experience: Moderate-High
  • ACR/NWNx4/MySQL host setup required
  • Priority level: Moderate-High (combat lag is a universal problem, though we're used to it by now)
Currently unassigned.

ALFA readable/writable Book GUI: Bartleby made a very impressive implementation of these several years back now, but it has never seen the light of day on the servers, primarily because it relies upon use of an XML file that will not read from inside haks, so it is nonfunctional unless we advocate players replacing base game resources inside their installs, potentially breaking future NWN2 patches/updates. It should be possible to edit this to just use DMFI letter replacement code instead of the custom language fonts. It is also possible there is a way to get the fonts recognized that I've overlooked. Even with the language support removed entirely, this would be a tremendous and long-overdue add to ALFA.
  • GUI scripting experience: Low-moderate (editing existing custom GUI)
  • NWN2 scripting experience: moderate
  • Priority level: Moderate (too long overdue, universally useful)
Currently unassigned.

Many more projects to be assigned. Do we have volunteers for any of the above? Glad to help anyone get started, all of these are things I've meant to do that just keep getting delayed by more pressing issues.
Hialmar
Fionn In Disguise
Posts: 3784
Joined: Sat Jan 03, 2004 11:54 am
Location: Toulouse, France
Contact:

Re: New ACR Projects...

Post by Hialmar »

I can work on "Portalling Improvements" if you want.
User avatar
AcadiusLost
Chosen of Forumamus, God of Forums
Posts: 5061
Joined: Tue Oct 19, 2004 8:38 am
Location: Montara, CA [GMT -8]
Contact:

Re: New ACR Projects...

Post by AcadiusLost »

Hialmar wrote:I can work on "Portalling Improvements" if you want.
If you like; though since I wrote the code for the portalling scripts we have now (and since I'm not able to contribute in any significant way to the actual plugin design/editing), I'd hoped this part would be something I or a volunteer might be able to take care of instead.
Hialmar
Fionn In Disguise
Posts: 3784
Joined: Sat Jan 03, 2004 11:54 am
Location: Toulouse, France
Contact:

Re: New ACR Projects...

Post by Hialmar »

*nods* we'll see who finishes first.
Basically it will depend on the decision to go "master server" or not for the vault.
But even if we decide to use a "master server" most of the code is already developed.
User avatar
Wynna
Dungeon Master
Posts: 5734
Joined: Sat Jan 03, 2004 10:09 am
Location: Seattle, WA (PST)

Re: New ACR Projects...

Post by Wynna »

I'm a huge fan of Bartleby's read/write books system. I make no promises of timeliness or even deliverability, but I'd like to give it another lookover. If all it takes is editing in DMFI code in place of custom fonts, I can see if I can make heads or tails of it.
Enjoy the game
Ronan
Dungeon Master
Posts: 4611
Joined: Sun Feb 20, 2005 9:48 am

Re: New ACR Projects...

Post by Ronan »

AcadiusLost wrote:Combat data logging optimization: Presently during each OnDeath() event of an NPC or creature killed by PCs or their associates, a flurry of records is pushed across to the remote SQL database documenting how much XP was earned by each PC in the party, who the party leader was, and what was killed. In a large-party situation with multiple opponents, this can lead to significant lagspikes at critical moments. It may be possible to store combat records as local variables on a dynamically created waypoint in the area until combat is finished, then push them out onto the database after confirming no one onserver is currently in combat. As an aside, such a system could also facilitate later development of a system to award XP based on Encounter Level, rather than the individual CR of each monster. Also possible there may be improvements that can be made to how the log records are sent through the MySQL plugin such that they have less overhead for the nwn2server (or can be sent collectively as a transaction instead of individually as separate DB accesses).
  • NWN2 scripting experience: Moderate-High
  • ACR/NWNx4/MySQL host setup required
  • Priority level: Moderate-High (combat lag is a universal problem, though we're used to it by now)
Currently unassigned.
I didn't see this post, but talked with AL on IRC. I implemented this today in a function called ACR_AsyncSQLQuery. I used a FIFO queue to feed a DelayCommand service which passes the queue elements to the db. Of course since nwscript is single-threaded this spreads the lag out rather than actually eliminating it. The recursive DelayCommand call is the first statement in the function, so failure of the service is extremely unlikely (you'd need that exact command to overflow the stack, which means you've already got problems). I was thinking of adding something to insure the service keeps running but I don't think its necessary.

We may wish to add an explicit timestamp to the above reporting if we don't have one already.

A better solution may be to alter the NWNx4 plug-in so NWNXSetString returns faster - if thats even possible. I wouldn't mind tackling this if its is what needs to be done.

I also created a generic LocalQueueString data structure if anyone has need of it. It should be pretty efficient since it keeps track of the head and tail (so everything is O(n) thanks to the nature of LocalVars...).

I haven't tested it yet as I don't have all the expansions I need, though I wrote unit tests.

The ideal solution may be to write a MongoDB plugin for NWNx4. Mongo is extremely fast, schema-less, and more intuitive for normal people to use (since it favors embedding data in other data) which I think works out very well for our purposes. However this may be total overkill - I've yet to even get in-game.
Ronan
Dungeon Master
Posts: 4611
Joined: Sun Feb 20, 2005 9:48 am

Re: New ACR Projects...

Post by Ronan »

The C++ code being executed every time a creature dies:
http://nwn.virusman.ru/svn/nwnx4/trunk- ... /mysql.cpp

Code: Select all

bool MySQL::Execute(char* query)
{
	MYSQL_RES *newResult;

	if (!connection)
	{
		if (!Reconnect())
		{
			wxLogMessage(wxT("! Error: Not connected."));
			return FALSE;
		}
	}

	// eat any leftover resultsets so mysql does not get out of sync
	while (mysql_more_results(&mysql))
	{
		mysql_next_result(&mysql);
		newResult = mysql_store_result(&mysql);
		mysql_free_result(newResult);
	}

	// execute the query
	if (logLevel == 2)
		wxLogMessage(wxT("* Executing: %s"), query);
	if (mysql_query(connection, (const char *)query) != 0)
	{
		unsigned int error_no = mysql_errno(&mysql);
		if (logLevel > 0)
				wxLogMessage(wxT("! SQL Error: %s (%lu)."), mysql_error(&mysql), error_no);

		// throw away last resultset if a SELECT statement failed
		if (_strnicmp(query, wxT("SELECT"), 6) == 0)
		{
			mysql_free_result(result);
			result = NULL;
			row = NULL;
			num_fields = 0;
		}

		if ((error_no == CR_SERVER_GONE_ERROR) && (Reconnect()))
		{
			// Retry query with new connection
			if (mysql_query(connection, (const char *)query) != 0)
				return FALSE;
		}
		else
		{
			return FALSE;
		}
	}

	// store the resultset in local memory
	newResult = mysql_store_result(&mysql);
	if (newResult == NULL)
	{
		if (mysql_field_count(&mysql) != 0)
		{
			// SELECT with an empty result set
			wxLogTrace(TRACE_VERBOSE, wxT("* Retrieved an empty resultset (mysql_query)"));
			mysql_free_result(result);
			result = NULL;
			row = NULL;
			num_fields = 0;
		}
		else
		{
			// NOT a SELECT like command
			wxLogTrace(TRACE_VERBOSE, wxT("* Retrieved an invalid resultset (NO SELECT) (mysql_query)"));

			// Try to advance to a non-empty resultset, if there is one.
			// This allows for calls to SQLFetch() to succeed automatically, 
			// even if the first n resultsets are empty or not valid.
			newResult = AdvanceToNextValidResultset();
			if (newResult)
			{
				mysql_free_result(result);
				result = newResult;
				row = NULL;
				num_fields = mysql_num_fields(result);
			}
		}

		if (mysql_errno(&mysql) != 0)
		{
			if (logLevel > 0)
				wxLogMessage(wxT("! Error (mysql_store_result): %s"), mysql_error(&mysql));
			return FALSE;	
		}
	}
	else
	{
		// successfully retrieved the resultset
		mysql_free_result(result);
		result = newResult;
		row = NULL;
		num_fields = mysql_num_fields(result);
	}
	
	return TRUE;
}
Clearly its passing the query then retrieving the result, blocking the whole time.

It looks like query results might be kept in a stack, so the best thing to do may be to create an AsyncExecute and pass Execute to a thread pool of some sort. Its probably been 10 years since I've looked at an C++. Anyone have any suggestions? I'll ask on the NWNx4 forums too.

I'm having trouble getting a dev environment setup,so it might take me a while to get this done.
Ronan
Dungeon Master
Posts: 4611
Joined: Sun Feb 20, 2005 9:48 am

Re: New ACR Projects...

Post by Ronan »

Acadius, or someone else with a test environment set up,

Can you try using "INSERT DELAYED" instead of "INSERT" for log (and possibly other) writes? If this helps, I will coalesce log writes which happen in quick succession and call it a day.

The thread is here:
http://www.nwnx.org/phpBB2/viewtopic.php?t=1761

Unfortunately I have little experience with this sort of threading concurrency (everything I do now is actor-based) and less with C++, so I don't think I can implement a proper fix in a short amount of time, unless I find some libraries that'll do the hard work for me. Testing concurrency tends to suck because everything often just hangs when it doesn't work.

Thanks.
User avatar
CloudDancing
Ancient Red Dragon
Posts: 2847
Joined: Sun Jan 03, 2010 6:31 am
Location: Oklahoma
Contact:

Re: New ACR Projects...

Post by CloudDancing »

Apparently Paazin has the XPcraft up and working on BG. News to me and to you.
Zelknolf
Chosen of Forumamus, God of Forums
Posts: 6139
Joined: Tue Jul 05, 2005 7:04 pm

Re: New ACR Projects...

Post by Zelknolf »

Cloud_Dancing wrote:Apparently Paazin has the XPcraft up and working on BG. News to me and to you.
It is only recently without any (known) gamebreaking bugs.

I am legendary in my capacity to make infinite gold from anything, you see, and that had to be tidied up. New code doesn't seem to make any assumptions based on equipment slots, which was the source of the initial bug. I'd probably defer to paazin as to whether or not enough testing has been done to pass it out to other servers; I know that I haven't done enough to the revision to stamp approved on it quite yet.
Ronan
Dungeon Master
Posts: 4611
Joined: Sun Feb 20, 2005 9:48 am

Re: New ACR Projects...

Post by Ronan »

AcadiusLost wrote:Combat data logging optimization.

This is done. Its far from ideal, but also far better than what we had before.

I am going to mess with travel map stuff next as I know BG and MS want this (and I think TSM will want some of me and Zelk's additions). I have the a population system done, but it needs a UI (and I hate UIs) so it will wait.
User avatar
Curmudgeon
Gadfly
Posts: 4312
Joined: Thu Oct 07, 2004 12:07 am
Location: East coast US

Re: New ACR Projects...

Post by Curmudgeon »

TSM wants, too.
- Curmudgeon
HDM ALFA 03 - The Silver Marches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Maxim #12: A soft answer turneth away wrath. Once wrath is looking the other way, shoot it in the head." - The Seventy Maxims of Maximally Effective Mercenaries

"This is not my circus. These are not my monkeys."

Realmslore: Daily Dwarf Common
User avatar
Brokenbone
Chosen of Forumamus, God of Forums
Posts: 5771
Joined: Mon May 16, 2005 1:07 am
Location: London, Ontario, Canada

Re: New ACR Projects...

Post by Brokenbone »

Since I couldn't find a thread about it, and it feels like a new-ish project, curious on the status of the Consumables Crafting stuff.

Lot of dead PCs lately = lot of new PCs = lot of folks who might like to bear in mind that wizards might be able to usefully scribe, or even lowbie clerics could brew useful potions, things of that nature.

Just like lowbies might remember that climb/swim etc. might be more than just RP skills before long, hah!
ALFA NWN2 PCs: Rhaggot of the Bruised-Eye, and Bamshogbo
ALFA NWN1 PC: Jacobim Foxmantle
ALFA NWN1 Dead PC: Jon Shieldjack

DMA Staff
Ronan
Dungeon Master
Posts: 4611
Joined: Sun Feb 20, 2005 9:48 am

Re: New ACR Projects...

Post by Ronan »

All the blueprints are made (or generated), about 500 of them I think. I'm working on the logic which assigns icons to them now.

The in-game crafting code is written (aside from some bits matching item props to spells for non-consumable DM-initiated crafting), but I may port it over to C#.

It'll be in the next hak release for sure.
Zelknolf
Chosen of Forumamus, God of Forums
Posts: 6139
Joined: Tue Jul 05, 2005 7:04 pm

Re: New ACR Projects...

Post by Zelknolf »

Indeed, such is totally posted on the front page, and I chortle merrily in your direction.
Locked