Articles written in Sadhana
Volume 45 All articles Published: 2 June 2020 Article ID 0141
The kernel of the modern operating system fails to ensure the authenticity of a running process while servicing a system call. Verifying the origin and integrity of a system call is an important security issue in terms of ensuring the proper functioning of an end-system. The conventional process identification parameterssuch as process identifier, process names and the executable flow exercised by the operating system are not reliable. As a result, a stealthy malware may mimic other processes to carry out many computer crimes, thus compromising the end-system. In this paper, we present a novel idea in which system call invocations made by a malicious application are verified during runtime in Windows operating system. To ensure the authenticity of a process while servicing a system call, we propose a behavior-based mechanism, namely, the process authenticationmechanism (PAM), for combating malicious code attacks that verifies the identity of each suspected process before being serviced by the kernel. The simulation and performance evaluation results confirm that our mechanism can effectively block all malicious samples that directly invoke system services in the kernel mode. PAM incurs no more than two percent overhead and helps to strengthen the overall system security.