Jump to content
AC Tool Forums
Sign in to follow this  
mysticdrew

Something I am working on.

Recommended Posts

Disclaimer: This is not released yet! -> Pending licensing, due to some dependencies that I require.

So one of the issues with AcTool, AutoIt, AutoHotkey, and likely even the new Divinity is that they are Windows specific. There are a few tools out there for MacOS and Linux but nothing that I have found works on all of them. Write one script/application and it is platform agnostic. 

I am in the process of writing a Java tool that will do just that! I am a java software engineer by trade so obviously it is language I am drawn too. I was thinking either java or python I went with java because I already had a basic project started for this tool. 

While there will be no "scripting engine" for this, it will be a dependency library of sorts with helpful methods, annotations, native tools to get the ball rolling right away. I may however write a scripting engine for this and it will use python or javascript/typescript. That is a long way off, if it ever happens.

I will post updates here as I progress on the project. It will be open source when I get my licensing situation figured out. One of the main things I want to provide with this project, is I want to make it easy enough that non programmers can use it, like they can with ACTool. ACTool is simple, yet powerful scripting tool but the syntax is not complex. It is what got me into my software development career. 

My current phase is creating a dynamic annotation system for events. (key events, mouse click events, custom events) 
Actool has an event processor for calling procedures below:

Procedure Foo when bar=foobar


I currently have a working key event system.  KeyEvent - KeyPressedEvent, KeyReleasedEvent that works like so.
 

    @SubscribeKeyEvent(key = "v", modifier = KeyModifier.CTRL_SHIFT, type=KeyEvent.Type.PRESSED)
    public static void onKeyPress()
    {
        // Fires on key press and key release
        // Fires when only the combination of CTRL+SHIFT+V is pressed/released (it is case agnostic)
    }

    @SubscribeKeyEvent(key = "b")
    public static void onKeyPressStatic(KeyEvent.KeyPressedEvent event)
    {
        // only fires when CTRL+SHIFT+V is pressed (it is case agnostic)
    }
    
    @SubscribeKeyEvent()
    public static void onKeyPressedEvent(KeyEvent.KeyReleasedEvent event) {
        // fires only when a key, any key, is released
        // validate it with
        if("a".equalsIgnoreCase(event.keyString)) {
            // a was pressed
        }
    }

Any combinations of qualifiers in the annotation or specific method parameter type will work. It's pretty robust. 

I am traveling for work the next week, I am not sure how much time I will have to work on it. 

However, until it is open sourced I have a couple slots on my github repo that I can provide access to a few users who wish to tinker or help with this. Please respond or PM me and we can chat. 

Future plans:

  • I will be adding a bunch of MouseEvents, the obvious click events, but also area events. When your mouse moves to an area, the event will execute and you can do stuff. 
  • Imaging tools, with workarounds for Windows10's slow getPixel garbage. 
  • Custom gradle plugin to make it easier to a project setup in Intellij or Eclipse
  • Guides and Documentation. 
  • And anything else that I can think of to make life easier.



Thoughts: I want to make it clear, I am not writing this to replace actool, divinity, autoit, or autohotkey. They all have their place, I just want something that is platform agnostic. This project is not new, I started it about 4 years ago and worked on it for a few days then got busy and forgot about it. I am picking it up again to hopefully get it workable and usable by a community.
 

Share this post


Link to post
Share on other sites

@mysticdrew Glad to see have something that you want to work on. And from what I see, it is a pretty good idea. I remember my Java days, I learned on v3.2 thru 4.0. I had even purchased a few Deitle & Deitle books back then to help me learn. I originally wanted to learn Java because of a game helper that I used for a OLD SCHOOL BBS game, turned multiplayer online game, that used Java for it's scripting engine. And I was dead set on learning Java to make sure that I could create the best scripts that I could for the game. Heck I even create a couple small, and minor, Java Web Applications. But, with the current version of Java I am completely lost.

 

From what I remember, Java was notoriously slow. Yes, it is fast for a Interpretive Language. But, it was slow when compared to non-Interpretive Languages. And that does not include the even slower start up time of the Java Application for which requires the Java engine to be loaded and running, the Java engine to then load the Java Application opcode at which point the Application is run. I assume that over the years that Jave engine has gotten faster. But, I still see every day the signs of how slow Java still is. Examples: All android devices OS is Pure Java. Chrome books, the Chrome OS is also Pure Java. And then there is Chrome itself, it is a Java Application.

I have not a thing against Java. It is a very wonderful language to work with. And it can do wonders when it comes to cross platform. Because it was made to be cross platform from the very beginning. (No other programming language can say that they are truly cross platform.) But, for certain applications where speed is critical. Java just does not have the ability to meet it. And as far as I know the there is only one thing slower than Java. And that is all Adobe programs/applications.

I know a little bit about Python and Javascript, never heard of Typescript. And I know that for a webpage application that Python and Javascript work wonders. But, again they are a interpretive language. And dependent on the languages engine/renderer to be loaded and run the code. Beyond what is used in various web sites or web pages. I don't know how fast these two script languages are.

 

And you are right about AC Tool and AutoIt being for Windows only. And as for "Divide" I have had plans on making it cross platform. The scripting engine being used is cross platform is already cross platform enabled, ie Linux, Android, MacOS, iOS, and a few others. And my programming environment can create cross platform programs/apps for all those operating systems as well. All I would have to do is create a cross platform GUI and take care of a few sections of code that is Windows specific.

But, my problem is this. I do not have any systems, other than Windows 7/8/8.1 and Android (phone) to test my work on. Granted I could go to any local Pawn shop and get two or three computers and put different Linux Distro's on them. But, my knowledge of Linux is: how to download the various distro's, install the distro from a Live CD/DVD, and run it from the GUI. That's it. I can't even install a program/app on it. And as for my knowledge of MacOS: Turn on computer, and then use. OH, and I do know that it is a custom version of some Linux distro. But, buying a used Mac system just to be able to test on, is so not going to happen. Even a used system would cost more than my whole setup now.

I like the Keyboard event and Mouse event idea. And the keyboard event is partially implemented in "Divide" already. And it would not take much to add what is missing. Plus, I was already been looking adding something quit similar, in another project that I have going. In that project you register Triggers, based on certain text being received from and being sent to the game within the data stream. This would include key presses being sent to the game. I had originally planed on expanding the keyboard portion and adding the mouse trigger. But, I have not worked on that project in years.

 

But, I do expect you to continue to work on your project. Because I will more than likely have use for what you create.

Share this post


Link to post
Share on other sites

Chrome is not written in Java, it is mostly C++ and Objective-C, On Android the Chrome UI bits are in java but the core is C++. 
Chrome OS, also C++. Andoid OS is mostly C++ but apps can be C++ or/and Java. It is widely thought that Android is pure java, it is not. The core is all C++ but the UI is a mix of java/c++. They expose the api through java, I think that is why most think it is a pure Java OS. 

While java has the perception from rumors of being slow it really is not. It probably was back in the 90's when it was first created. But on today's machines it can be as fast as native applications. 
I am using several libraries that give native system access without pulling the data from the JVM. All input /mouse/keys/pixel data are from the native system through C++ dlls. 
Yes there is a bit of slowness to startup in the IDE(havent tested the compiling), because I have to scan the classpath for annotations and build dynamic classes using ASM writing optcode instead of using reflection for my eventbus. Writing Optcode is a huge pain, but for speed, it's worth it. 

Many of the tools/macros/bots people have written in AcTool will be just as fast in Java. They are pretty small in the scope of things. At work our Enterprise systems are all java and they are blazing fast, huge applications, tons of threads, still extremely fast. 

I also plan on caching a lot to speed things up. For example, pixel detection is a heavy call for any language. My plan is when a getColor call is executed, it will take a small image around that pixel and store for maybe 10-50ms, then all calls in that area are done on that image. So if you want to scan maybe a healthbar, you loop through a getColor 100 times in a matter of nanoseconds. That can be a very heavy call if you check the pixel each time and it will take more than a few ns. The first getColor(x,y) call will check if that region has a living cache, if it does, it will scan cached image. If it does not, it will use a JNA call to create the region, toss it into cache and scan it. In theory it should be lighting fast. I have not implemented any of the color detection code yet, but that is my plan. 

If you want access to the github project, private message me your GH account name and I will add you. I think I can only add 3 people to free private GH projects. 

Share this post


Link to post
Share on other sites

@mysticdrew

Now as for the coming up with workaround for the getPixel problem. I will give you the following information that I learned many years ago.

 

Roughly back in 2013 a AC Tool user had posted information about a problem he was seeing with AC Tool and using the various Color commands. (That post is still here somewhere. Oh wait that was you.) At that time, and many many times since then. I have looked into the problem and for a way to get around it. So, when it comes to the GetPixel being slow. It is a problem created by Micro$ucks adding advance display features, like Aero and transparent boarders, to Vista. They refer to such features as composition.

Instead of going back and working on the code that does all the rendering of the display. And properly add in the required code to display these new display features, like Aero and the transparent borders, Micro$ucks decided to add a program that sits between the Windows display renderer and the display memory. It is called the Desktop Window Manager or DWM for short. What the DWM does is it intercepts all rendered display information going to, or being requested from, the display memory. Caches the rendered display information frame by frame. And through it's cached frame(s) performs it needs to for Windows to display the various advance display features. Then the DWM sends that new rendered display from its cache to the display memory to be displayed.

But, here is the rub of the problem. In return, when requesting information about the display and/or its rendering, such as using the GetPixel. All of the requests have to do through DWM. And it is DWM that supplies the requested information back to the caller, based on what it has in its cache of display frames. This should be pretty fast but it is not. From what I have found, DWM converts the cached frame to something like a bmp and gets the information from that and then sends it back.

It is during this conversion process that things become slow. And this cost is per call.  So, using the GetPixel in a tight loop causes the times to execute the command to increase.

 

All my research shows the only way to get around the cost comes to doing one of the following:

  • Stop using commands such as GetPixel in tight loops.
  • When scanning a display, like looking for a Object in AC Tool, use BitBlt and scanlines process (or something similar).
  • Turn of Aero effects, or composition, threw the Windows Theme. By using the Classic theme.
  • Use the DWM API to turn ofF DWM composition. With Windows 8 and newer, this is no longer an option.
  • Create your own display driver that access the display memory directly.

That's it, When it comes to AC Tool the only thing that can be done is to use the BitBlt and Sanline for the searching for a Object. But, there is nothing that can be done to speed up the IsColor or RGBColor commands. As they are checking a single pixel. I have already updated the CreateObject and IsObject commands in "Divide" with the BitBlt and scanlines option.

Now, the BitBlt and scanline process uses BitBlt to capture the screen, or portion there of, into a image object. Then you use the ScanLines to scan through the image for the pixel information. I had already thought of doing something similar with the IsColor and RGBColor commands instead of using the GetPixel as AC Tool currently uses. Capture the screen, or a portion of it, into a image object, get the individual pixel information, then release the image object. But, you would have the overhead of having to capture the screen and placing it in a image object each time you use the any of the IsColor or RGBColor commands. This can cause the use of the IsColor and RGBColor commands to be slow if used in a tight loop. And isn't that what is happening when using the GetPixel through DWM?

Now, I had thought about capturing the screen and placing it in a cached image object and then updating the cached image object every so often, like every so many ms. And then have the IsColor and RGBColor commands work with the cached image object. But, the problem comes down to how often do we update the cached image? Because when using the IsColor or RGBColor in a tight loop. The information those commands will be using will be old data until such time as the image object is updated. And then only the first call by a IsColor/RGBColor right after the image object is updated will be based on new, or current, data. So, how often do we need to update the image object. When will it start to effect the speed performance?

I hope that you can come up with an idea for this. Because when it comes to using tight loops of GetPixel, or its equivalent when using a cached image, is going to have a performance hit or used old data.

 

And maybe your programming skills is good enough to create a display driver. But if not then the only way to see speed in a tight loop of GetPixel commands is to turn off display composition through your desktop themes.

 

Share this post


Link to post
Share on other sites
7 hours ago, mysticdrew said:

Chrome is not written in Java, it is mostly C++ and Objective-C, On Android the Chrome UI bits are in java but the core is C++. 
Chrome OS, also C++. Andoid OS is mostly C++ but apps can be C++ or/and Java. It is widely thought that Android is pure java, it is not. The core is all C++ but the UI is a mix of java/c++. They expose the api through java, I think that is why most think it is a pure Java OS. 

While java has the perception from rumors of being slow it really is not. It probably was back in the 90's when it was first created. But on today's machines it can be as fast as native applications. 

I stand corrected.

7 hours ago, mysticdrew said:

While java has the perception from rumors of being slow it really is not. It probably was back in the 90's when it was first created. But on today's machines it can be as fast as native applications. 
I am using several libraries that give native system access without pulling the data from the JVM. All input /mouse/keys/pixel data are from the native system through C++ dlls. 
Yes there is a bit of slowness to startup in the IDE(havent tested the compiling), because I have to scan the classpath for annotations and build dynamic classes using ASM writing optcode instead of using reflection for my eventbus. Writing Optcode is a huge pain, but for speed, it's worth it. 

Many of the tools/macros/bots people have written in AcTool will be just as fast in Java. They are pretty small in the scope of things. At work our Enterprise systems are all java and they are blazing fast, huge applications, tons of threads, still extremely fast. 

As I stated before, it has been a very long time since I worked with Java. And I assumed that the JVM has gotten faster over the years. And it is great to hear that it has gotten a lot faster since v4.

7 hours ago, mysticdrew said:

I also plan on caching a lot to speed things up. For example, pixel detection is a heavy call for any language. My plan is when a getColor call is executed, it will take a small image around that pixel and store for maybe 10-50ms, then all calls in that area are done on that image. So if you want to scan maybe a healthbar, you loop through a getColor 100 times in a matter of nanoseconds. That can be a very heavy call if you check the pixel each time and it will take more than a few ns. The first getColor(x,y) call will check if that region has a living cache, if it does, it will scan cached image. If it does not, it will use a JNA call to create the region, toss it into cache and scan it. In theory it should be lighting fast. I have not implemented any of the color detection code yet, but that is my plan. 

Well, we seem to be thinking the same thing here. (Note: I had not read your response prior to doing my second post.) But, you can see my concerns about the cached image and the using of the IsColor/RGBColor or your getColor in a tight loop. Yes, you can increase the speed to doing 100 checks within a few ns. But, if you are only updating the image cache once every 10-50ms. Then 1 time out of thousands of iterations, between cache updates, will the check be based on current, or semi-current, pixel data. Basically you'd be spinning your wheels for nothing.

Basically, it does not matter of what we come up with to solve the problem. Using tight loops with the mentioned commands is no longer viable. All loops must include some form of delay between iterations. A delay of 5-10 ms is all that is needed. This will cut down on the performance hit. Or when using a image cache, it will cut down on the spinning your wheels for nothing.

 

Maybe, you can come of with something that works. Because I really want to increase the speed of the IsColor and RGBColor commands in "Divide." So, I fully give you notice that when you do. I will be basing what is used by IsColor and RGBColor on what you have come up with.

 

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use and to our Privacy Policy.