===================== Development and Build ===================== The MSR Profile services are an open-source project, with all code and changes referenced in our GIT repository. This is hosted at https://bitbucket.org/squarehost/msr-cmd-service We would love to field any bug requests, contributions and/or pull requests from other users or developers so please get involved. Our discord channel can be found at https://discord.gg/pDXsgF for general conversation and help. ---------------------------------- Creating a Development Environment ---------------------------------- It is very straightforward to create your own development environment if you wish to experiment with the services, develop or debug them locally. Use the following steps to get started. .. topic:: Suitable Users Note, whilst this is quite a lightweight and straightforward development project, it is still only intended for experienced Python developers to follow these steps. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Step 1: Download a recent version of Python 3 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ At the time of writing the services are written using `Python 3.9 `_. Whilst it is likely the codebase will be compatible with other versions, it is, of course, recommended to download this version. Download the installer, and follow the relevant install instructions for your platform. It is recommended to install system-wide, but you can install locally, just substitute the **python** commands below for the Python executable install location. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Step 2: Create a parent folder for the project ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This will be a top-level folder containing all the files needed for our project. You are free to create this how and where you like. Within this example we are going to be using :code:`D:\work\msr`, but substitute this with whatever path is suitable for your environment. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Step 3: Clone the development codebase ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ensuring that the `GIT `_ system is installed on your machine, issue the following commands: .. code-block:: batch cd D:\work\msr git clone https://bitbucket.org/squarehost/msr-cmd-service.git This will clone the relevant GIT environment into :code:`D:\work\msr`. This is a standard GIT environment, so if you do wish to contribute code or changes to the project, we would recommend you create your clone of the project and submit a pull request accordingly. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Step 4a: Create a virtual environment for libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Whilst this is technically optional, it is highly recommended to set-up a virtual environment to prevent any installed Python package and libraries isolated to this specific project, preventing other installed Python-based software from interfering with libraries required for the MSR system: .. code-block:: batch cd D:\Work\msr python -m venv env-py This should create a folder called **env-py** within :code:`D:\work\msr`, which contains a basic Python virtual environment. Any time you wish to perform install, debug or build operations for the MSR services, you should ensure your virtual environment is activated: .. code-block:: batch env-py\scripts\activate You will see the terminal prompt update to include the environment name accordingly. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Step 4b: Install Required Python Libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ensure your virtual environment is activate (if relevant), and install the dependencies for the service/app you wish to work on. If in any doubt, just run all below commands to install all required dependencies as they will not conflict: .. code-block:: batch cd D:\Work\msr pip install -r msr-cmd-service\msr_win32_svc\pip-requirements.txt pip install -r msr-cmd-service\profile_win32_svc\pip-requirements.txt pip install -r msr-cmd-service\taskbar_menu\msr_taskbar\pip-requirements.txt ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Step 5: Install Required WinRing0 MSR DLL ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ These services use the WinRing0x64.dll from http://openlibsys.org. You should obtain the 64-bit DLLs from their `Github Repo `_ and either build these yourself or download the 1.3.0 release from the relevant link. Place the WinRing0x64.dll and WinRing0x64.sys driver within your :code:`C:\Windows\System32` folder. ---------------------------------- Running services in dev/debug mode ---------------------------------- As part of the development process, you should run the services in debug mode. Any changes you make to the code will be applied without having to go through the complete build/install process. In addition, when running in debug mode, a service will run in the foreground command prompt window you utilise, and issue more detailed debugging information to the console window. .. topic:: Security of Services in Debug Mode Note, the security of the MSR service named pipe is relaxed in debug mode to allow **any local system users** to access the pipe and send command messages. When installed in production (e.g. through the installer) and not running in debug mode, the pipe is only accessible to the **Local System** account, which effectively allows only other system service to access it, and not other users. As the debug services are run locally from the command prompt, this restriction has to be relaxed to allow the services to communicate. Whilst this is all handled automatically when running in debug mode, this is obviously not suitable for anything other than temporary development due to the potential security issues with open access to the control pipe. .. topic:: Clashing with production services When running a service in debug mode it still uses the Windows Service Framework to operate. If you are running the production services, it will clash with these so you should issue the following commands to stop any running services: .. code-block:: batch net stop winmax-msr-tdpuv-svc net stop winmax-msr-web-svc **Note**, this will likely require administrator access. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. Running the MSR Service in debug mode ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To run the MSR service (which handles actual communication with the underlying registers and subsystems), open a Powershell prompt **as the administrator user (Run as Administrator)**, and issue the following commands: .. code-block:: batch cd D:\work\msr # Activate virtual environment if applicable env-py\scripts\activate cd msr-cmd-service\msr_win32_svc # Only required if you have not previously run the service in debug # mode, or have run the production installer since. python service.py --startup=manual install python service.py debug The service should begin, and you should see it attach to the relevant named pipe and wait for commands from the profile/web service. If you see a long stream of messages similar to :code:`(5, 'CreateNamedPipe', 'Access is denied.')` it means another instance of the service is running. This could be the installed production service - if so, press ctrl+C to exit the debug service and stop the running MSR service using the services control panel or the commands listed in the above **Clashing with production** topic. .. image:: ../img/dev/debug-output.png :width: 600 :alt: The MSR service running correctly in debug mode **To exit a running debug service, press the ctrl+C key combination.** ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2. Running the Web/Profile Service in Debug Mode ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To run the web/profile service, open a Powershell prompt **as the administrator user (Run as Administrator)**, and issue the following commands: .. code-block:: batch cd D:\work\msr # Activate virtual environment if applicable env-py\scripts\activate cd msr-cmd-service\profile_win32_svc\profile_svc\msr_web # Only required if you have not previously run the service in debug # mode, or have run the production installer since. python service.py --startup=manual install python service.py debug You should see the service run through some basic initial tasks (such as creating and migrating a development database), before logging a messages indicating the service is running:: MSR running using version 3.1.2, using settings 'msr_web.settings' Starting server at http://127.0.0.1:8998/ The listener threads (responsible for handling automation) will also log various commands to the console, along with any web requests and general log messages. **To exit a running debug service, press the ctrl+C key combination.** .. topic:: The development database and log files. When running in production mode, the database storage for all profiles, and the log files, are stored within the Local System user profile AppData folder, by default :code:`C:\Windows\System32\config\systemprofile\AppData\Local` When running in debug mode, the service utilises your local user's profile service for storage - meaning it maintains a separate copy of the database and log files just for debug mode. For example, my local user AppData folder is :code:`C:\Users\dev\AppData\Local` This means that any data you have created in production, such as profiles, user passwords, profile history and so on will not transfer to the debug service, and vice versa. Whilst we recommend this approach for obvious reasons, if you do wish to copy the existing data from production to give you a starting point, you can manually copy the following database (and log files if required) from the :code:`systemprofile\AppData\Local` folder to the local user :code:`AppData\Local` folder. * *msr-web.sqlite3*: Main database file with all profiles and settings It is not neccessary, but you can also copy the log files too: * *msr-web.log*: Most recent log file for the web/profile service * *msr-service.log*: Most recent log file for the MSR service Ensure you stop and restart the debug Web/Profile service (Ctrl+C) before copying the files. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 3. Running the Taskbar App in Debug Mode ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To run the web/profile service, open a Powershell prompt, and issue the following commands: .. code-block:: batch cd D:\work\msr # Activate virtual environment if applicable env-py\scripts\activate cd msr-cmd-service\taskbar_menu\msr_taskbar python taskbar.py **Note,** to exit the taskbar app, you must use the exit command located on the tray menu itself - Ctrl+C will have no effect. ---------------------------- Creating a Build Environment ---------------------------- As part of development/testing you may wish to build the services into bundled executables, which themselves can then be executed in debug mode. The bundled executables contain all the required dependencies, automatically extracted from the virtual environment, and can therefore be run in a "stand-alone" fashion. Building the bundled executables is also very straightforward, but does require some additional dependencies. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Install Required Python Build Libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Ensure your virtual environment is activated, and issue the following command:: pip install -r msr-cmd-service\build_scripts\build-requirements.txt This will download and install the Python packages to both build the executables (`PyInstaller `_) and documentation. ^^^^^^^^^^^^^^^^^^^^^^^^ Copy WinRing0 DLL/Driver ^^^^^^^^^^^^^^^^^^^^^^^^ You should copy the WinRing0x64.dll and WinRing0x64.sys files into the :code:`msr-cmd-service\build_scripts\msr` folder. These DLLs are bundled into the executable by the build scripts. ^^^^^^^^^^^^^^^^^^^^^ Execute Build Scripts ^^^^^^^^^^^^^^^^^^^^^ The build process is handled by some basic automated build scripts, located in the :code:`msr-cmd-service\build_scripts` folder. Resulting distribution files can be found in the :code:`\msr-cmd-service\dist` folder, which will be created during the build process. Services can be built using two main methods: ````````````````````````````````` 1. Build individual services/apps ````````````````````````````````` You may build the MSR, web/profile, and taskbar apps individually. This is useful if you are just testing changes to an individual service. A separate folder will be created within the *dist* folder for each service/ app, including all dependencies. For the MSR and web/profile services, these can then be installed as system-wide services, or debugged using the debug API call, using the following commands: .. code-block:: batch cd D:\work\msr\dist\[service_folder] ; To install as an auto-startup system service [service_exe.exe] --startup=auto install ; To start the service [service_exe.exe] start ; Or to debug... [service_exe.exe] --startup=manual install ; To start the service [service_exe.exe] debug Replace [service_folder] and [service_exe] with msr_svc or msr_profile_svc accordingly. **Note**, again, these services will clash with any existing installed production services if they are active, so stop these using the same method as described in the previous debug section. The taskbar app can be run directly, in the :code:`dist\msr_taskbar` folder, by executing *msr_taskbar.exe*. To build the services as described, execute the relevant script within the *build_scripts* folder. Ensure your virtual environment is activated prior to executing these. * *build_msr_svc.bat*: Builds the MSR service (:code:`msr_svc\msr_svc.exe`) * *build_profile_svc.bat*: Builds the web/profile service (:code:`msr_profile_svc\msr_profile_svc.exe`) * *build_msr_taskbar.bat*: Builds the tray app (:code:`msr_taskbar\msr_taskbar.exe`) `````````````````````````````` 2. Build bundled services/apps `````````````````````````````` The two services and the taskbar app can also be built as a bundle. This means they share common resources, and as such take up less space on disk. This is only required if you wish to generate an install package (see the following section) in order to install and test production services. To do this, execute the :code:`build_scripts\build_msr_bundle.bat` file. This will build the documentation, and all three individual projects, and copy the relevant files to the :code:`dist\_bundle` folder. **Note, you cannot run these executables directly as per the individual builds as the files need to be copied into the main folder (_bundle) where the relevant resources are contained.** ------------------------------- Building a Deployment Installer ------------------------------- The project also contains basic configuration and scripts to allow a fully automated, single-file installer to be created. This is how the main builds are distributed, and is created using the `Nullsoft Scriptable Install System (NSIS) `_. The installers contain all neccessary dependencies, and allow the services and task bar app to be installed on machines that do not have any of the development environment requirements installed. In effect, this produces a production/deployment installer. It is useful to be able to create these installers during development or testing, as it allows the production process to be tested with any changes you may have made, across a variety of different machines without the hassle of installing and replicating development environments. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Obtain the NSIS Build System ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Download the latest version of NSIS from `the project homepage `_. At the time of writing the system uses v3.06.1. Install NSIS using the installer. No additional plugins are required currently. ^^^^^^^^^^^^^^^^^^^^^^^^ Build the NSIS Installer ^^^^^^^^^^^^^^^^^^^^^^^^ The NSIS installer files are located within the :code:`msr-cmd-service\installer` folder. The main installer/uninstaller file itself is contained within the *interactive.nsi* file. It is recommended that you change a few variables within this file when creating temporary builds, to ensure they can be easily distinguished from actual builds, for example .. code-block:: nsis !define MSR_VERSION "0.9.9.myversion" OutFile "InstallMSR${MSR_VERSION}_test.exe" This will ensure that the outputted installer has both a filename and banner message indicating this is a test build. To build the installer itself, **ensure that you have created bundles services/apps following the instructions within the previous section**, and then launch NSIS. Select *Compile NSIS Scripts*, and then *File* -> *Load Script*, navigating to the :code:`msr-cmd-service\installer\interactive.nsi` file. NSIS should then proceed to add all the build files, and create an automated installer. The installer will be located in the :code:`msr-cmd-service\installer` folder, given the name defined in the *OutFile* setting above. This installer can then be installed directly, or copied to another machine for testing purposes. **Note**, the services will be re-installed, so any services you have configured for debugging within the previous sections will need to be "re-installed" (e.g. with *service.py install*) or you will find they error when service.[py|exe] debug is run.