Development and Build¶
The MSR Profile services are an open-source project, with all code and changes referenced in our GIT repository. This is hosted at https://bitbucket.org/squarehost/msr-cmd-service
We would love to field any bug requests, contributions and/or pull requests from other users or developers so please get involved. Our discord channel can be found at https://discord.gg/pDXsgF for general conversation and help.
Creating a Development Environment¶
It is very straightforward to create your own development environment if you wish to experiment with the services, develop or debug them locally. Use the following steps to get started.
Suitable Users
Note, whilst this is quite a lightweight and straightforward development project, it is still only intended for experienced Python developers to follow these steps.
Step 1: Download a recent version of Python 3¶
At the time of writing the services are written using Python 3.9. Whilst it is likely the codebase will be compatible with other versions, it is, of course, recommended to download this version.
Download the installer, and follow the relevant install instructions for your platform. It is recommended to install system-wide, but you can install locally, just substitute the python commands below for the Python executable install location.
Step 2: Create a parent folder for the project¶
This will be a top-level folder containing all the files needed for our
project. You are free to create this how and where you like.
Within this example we are going to be using D:\work\msr, but substitute
this with whatever path is suitable for your environment.
Step 3: Clone the development codebase¶
Ensuring that the GIT system is installed on your machine, issue the following commands:
cd D:\work\msr
git clone https://bitbucket.org/squarehost/msr-cmd-service.git
This will clone the relevant GIT environment into
D:\work\msr. This is a standard GIT environment, so if you do
wish to contribute code or changes to the project, we would recommend
you create your clone of the project and submit a pull request accordingly.
Step 4a: Create a virtual environment for libraries¶
Whilst this is technically optional, it is highly recommended to set-up a virtual environment to prevent any installed Python package and libraries isolated to this specific project, preventing other installed Python-based software from interfering with libraries required for the MSR system:
cd D:\Work\msr
python -m venv env-py
This should create a folder called env-py within D:\work\msr, which
contains a basic Python virtual environment. Any time you wish to perform
install, debug or build operations for the MSR services, you should ensure your
virtual environment is activated:
env-py\scripts\activate
You will see the terminal prompt update to include the environment name accordingly.
Step 4b: Install Required Python Libraries¶
Ensure your virtual environment is activate (if relevant), and install the dependencies for the service/app you wish to work on. If in any doubt, just run all below commands to install all required dependencies as they will not conflict:
cd D:\Work\msr
pip install -r msr-cmd-service\msr_win32_svc\pip-requirements.txt
pip install -r msr-cmd-service\profile_win32_svc\pip-requirements.txt
pip install -r msr-cmd-service\taskbar_menu\msr_taskbar\pip-requirements.txt
Step 5: Install Required WinRing0 MSR DLL¶
These services use the WinRing0x64.dll from http://openlibsys.org. You should obtain the 64-bit DLLs from their Github Repo and either build these yourself or download the 1.3.0 release from the relevant link.
Place the WinRing0x64.dll and WinRing0x64.sys driver within your
C:\Windows\System32 folder.
Running services in dev/debug mode¶
As part of the development process, you should run the services in debug mode. Any changes you make to the code will be applied without having to go through the complete build/install process. In addition, when running in debug mode, a service will run in the foreground command prompt window you utilise, and issue more detailed debugging information to the console window.
Security of Services in Debug Mode
Note, the security of the MSR service named pipe is relaxed in debug mode to allow any local system users to access the pipe and send command messages.
When installed in production (e.g. through the installer) and not running in debug mode, the pipe is only accessible to the Local System account, which effectively allows only other system service to access it, and not other users.
As the debug services are run locally from the command prompt, this restriction has to be relaxed to allow the services to communicate. Whilst this is all handled automatically when running in debug mode, this is obviously not suitable for anything other than temporary development due to the potential security issues with open access to the control pipe.
Clashing with production services
When running a service in debug mode it still uses the Windows Service Framework to operate. If you are running the production services, it will clash with these so you should issue the following commands to stop any running services:
net stop winmax-msr-tdpuv-svc
net stop winmax-msr-web-svc
Note, this will likely require administrator access.
1. Running the MSR Service in debug mode¶
To run the MSR service (which handles actual communication with the underlying registers and subsystems), open a Powershell prompt as the administrator user (Run as Administrator), and issue the following commands:
cd D:\work\msr
# Activate virtual environment if applicable
env-py\scripts\activate
cd msr-cmd-service\msr_win32_svc
# Only required if you have not previously run the service in debug
# mode, or have run the production installer since.
python service.py --startup=manual install
python service.py debug
The service should begin, and you should see it attach to the relevant
named pipe and wait for commands from the profile/web service. If you
see a long stream of messages similar to (5, 'CreateNamedPipe', 'Access is denied.')
it means another instance of the service is running. This could be
the installed production service - if so, press ctrl+C to exit the
debug service and stop the running MSR service using the services control
panel or the commands listed in the above Clashing with production topic.
To exit a running debug service, press the ctrl+C key combination.
2. Running the Web/Profile Service in Debug Mode¶
To run the web/profile service, open a Powershell prompt as the administrator user (Run as Administrator), and issue the following commands:
cd D:\work\msr
# Activate virtual environment if applicable
env-py\scripts\activate
cd msr-cmd-service\profile_win32_svc\profile_svc\msr_web
# Only required if you have not previously run the service in debug
# mode, or have run the production installer since.
python service.py --startup=manual install
python service.py debug
You should see the service run through some basic initial tasks (such as creating and migrating a development database), before logging a messages indicating the service is running:
MSR running using version 3.1.2, using settings 'msr_web.settings'
Starting server at http://127.0.0.1:8998/
The listener threads (responsible for handling automation) will also log various commands to the console, along with any web requests and general log messages.
To exit a running debug service, press the ctrl+C key combination.
The development database and log files.
When running in production mode, the database storage for all profiles,
and the log files, are stored within the Local System user profile AppData
folder, by default C:\Windows\System32\config\systemprofile\AppData\Local
When running in debug mode, the service utilises your local user’s
profile service for storage - meaning it maintains a separate copy
of the database and log files just for debug mode. For example, my
local user AppData folder is C:\Users\dev\AppData\Local
This means that any data you have created in production, such as profiles, user passwords, profile history and so on will not transfer to the debug service, and vice versa.
Whilst we recommend this approach for obvious reasons, if you do wish
to copy the existing data from production to give you a starting point,
you can manually copy the following database (and log files if required)
from the systemprofile\AppData\Local folder to the local user
AppData\Local folder.
msr-web.sqlite3: Main database file with all profiles and settings
It is not neccessary, but you can also copy the log files too:
msr-web.log: Most recent log file for the web/profile service
msr-service.log: Most recent log file for the MSR service
Ensure you stop and restart the debug Web/Profile service (Ctrl+C) before copying the files.
3. Running the Taskbar App in Debug Mode¶
To run the web/profile service, open a Powershell prompt, and issue the following commands:
cd D:\work\msr
# Activate virtual environment if applicable
env-py\scripts\activate
cd msr-cmd-service\taskbar_menu\msr_taskbar
python taskbar.py
Note, to exit the taskbar app, you must use the exit command located on the tray menu itself - Ctrl+C will have no effect.
Creating a Build Environment¶
As part of development/testing you may wish to build the services into bundled executables, which themselves can then be executed in debug mode. The bundled executables contain all the required dependencies, automatically extracted from the virtual environment, and can therefore be run in a “stand-alone” fashion.
Building the bundled executables is also very straightforward, but does require some additional dependencies.
Install Required Python Build Libraries¶
Ensure your virtual environment is activated, and issue the following command:
pip install -r msr-cmd-service\build_scripts\build-requirements.txt
This will download and install the Python packages to both build the executables (PyInstaller) and documentation.
Copy WinRing0 DLL/Driver¶
You should copy the WinRing0x64.dll and WinRing0x64.sys files into the
msr-cmd-service\build_scripts\msr folder.
These DLLs are bundled into the executable by the build scripts.
Execute Build Scripts¶
The build process is handled by some basic automated build scripts,
located in the msr-cmd-service\build_scripts folder. Resulting
distribution files can be found in the \msr-cmd-service\dist folder,
which will be created during the build process.
Services can be built using two main methods:
1. Build individual services/apps¶
You may build the MSR, web/profile, and taskbar apps individually. This is useful if you are just testing changes to an individual service. A separate folder will be created within the dist folder for each service/ app, including all dependencies.
For the MSR and web/profile services, these can then be installed as system-wide services, or debugged using the debug API call, using the following commands:
cd D:\work\msr\dist\[service_folder]
; To install as an auto-startup system service
[service_exe.exe] --startup=auto install
; To start the service
[service_exe.exe] start
; Or to debug...
[service_exe.exe] --startup=manual install
; To start the service
[service_exe.exe] debug
Replace [service_folder] and [service_exe] with msr_svc or msr_profile_svc accordingly. Note, again, these services will clash with any existing installed production services if they are active, so stop these using the same method as described in the previous debug section.
The taskbar app can be run directly, in the dist\msr_taskbar folder,
by executing msr_taskbar.exe.
To build the services as described, execute the relevant script within the build_scripts folder. Ensure your virtual environment is activated prior to executing these.
build_msr_svc.bat: Builds the MSR service (
msr_svc\msr_svc.exe)build_profile_svc.bat: Builds the web/profile service (
msr_profile_svc\msr_profile_svc.exe)build_msr_taskbar.bat: Builds the tray app (
msr_taskbar\msr_taskbar.exe)
2. Build bundled services/apps¶
The two services and the taskbar app can also be built as a bundle. This means they share common resources, and as such take up less space on disk.
This is only required if you wish to generate an install package (see the following section) in order to install and test production services.
To do this, execute the build_scripts\build_msr_bundle.bat file.
This will build the documentation, and all three individual projects, and
copy the relevant files to the dist\_bundle folder.
Note, you cannot run these executables directly as per the individual builds as the files need to be copied into the main folder (_bundle) where the relevant resources are contained.
Building a Deployment Installer¶
The project also contains basic configuration and scripts to allow a fully automated, single-file installer to be created. This is how the main builds are distributed, and is created using the Nullsoft Scriptable Install System (NSIS).
The installers contain all neccessary dependencies, and allow the services and task bar app to be installed on machines that do not have any of the development environment requirements installed. In effect, this produces a production/deployment installer.
It is useful to be able to create these installers during development or testing, as it allows the production process to be tested with any changes you may have made, across a variety of different machines without the hassle of installing and replicating development environments.
Obtain the NSIS Build System¶
Download the latest version of NSIS from the project homepage. At the time of writing the system uses v3.06.1.
Install NSIS using the installer. No additional plugins are required currently.
Build the NSIS Installer¶
The NSIS installer files are located within the msr-cmd-service\installer folder.
The main installer/uninstaller file itself is contained within the
interactive.nsi file.
It is recommended that you change a few variables within this file when creating temporary builds, to ensure they can be easily distinguished from actual builds, for example
!define MSR_VERSION "0.9.9.myversion"
OutFile "InstallMSR${MSR_VERSION}_test.exe"
This will ensure that the outputted installer has both a filename and banner message indicating this is a test build.
To build the installer itself, ensure that you have created bundles services/apps following the instructions within the previous section, and then launch NSIS.
Select Compile NSIS Scripts, and then File -> Load Script, navigating
to the msr-cmd-service\installer\interactive.nsi file. NSIS should
then proceed to add all the build files, and create an automated installer.
The installer will be located in the msr-cmd-service\installer
folder, given the name defined in the OutFile setting above.
This installer can then be installed directly, or copied to another machine for testing purposes. Note, the services will be re-installed, so any services you have configured for debugging within the previous sections will need to be “re-installed” (e.g. with service.py install) or you will find they error when service.[py|exe] debug is run.