背景介紹
一套在RHEL的集羣上運行的Oracle實例,是用systemd服務啓動Oracle實例的(方便集羣的切換操作).在測試過程中發現標準大頁沒有被用上.
具體情況如下所示:
$ grep HugePages /proc/meminfo
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 2034
HugePages_Free: 2034
HugePages_Rsvd: 0
HugePages_Surp: 0
原因分析
如上所示,HugePages_Free的值為2034, HugePages_Total的值也是2034,也就是説標準大頁完全沒有使用.
Linux服務器的基本信息如下
$ more /etc/redhat-release
Red Hat Enterprise Linux release 8.10 (Ootpa)
$ free -m
total used free shared buff/cache available
Mem: 11697 4929 5986 17 780 6612
Swap: 16383 0 16383
檢查數據庫的參數如下, 完全符合條件:
SQL> select banner from v$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
SQL> col name for a20;
SQL> col value for a32;
SQL> select name, value from v$parameter
2 where name in ('memory_target','sga_target','use_large_pages');
NAME VALUE
-------------------- --------------------------------
use_large_pages TRUE
sga_target 4261412864
memory_target 0
SQL>
內核參數vm.nr_hugepages也是正確設置的,具體如下所示
$ grep vm.nr_hugepages /etc/sysctl.conf
vm.nr_hugepages = 2034
$ ./hugepages_settings.sh
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
* For ASM instance, it needs to configure ASMM instead of AMM.
* The 'pga_aggregate_target' is outside the SGA and
you should accommodate this while calculating the overall size.
* In case you changes the DB SGA size,
as the new SGA will not fit in the previous HugePages configuration,
it had better disable the whole HugePages,
start the DB with new SGA size and run the script again.
And make sure that:
* Oracle Database instance(s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
(See Doc ID 749851.1)
* The shared memory segments can be listed by command:
# ipcs -m
Press Enter to proceed...
Recommended setting: vm.nr_hugepages = 2034
資源限制配置文件limits.conf中的memlock的設置也是正確的.如下所示:
# grep memlock /etc/security/limits.conf
# - memlock - max locked-in-memory address space (KB)
oracle soft memlock 10485760
oracle hard memlock 10485760
# su - oracle
Last login: Fri Aug 8 13:54:36 CST 2025 on pts/0
$ ulimit -l
10485760
$ grep memlock /etc/security/limits.conf
# - memlock - max locked-in-memory address space (KB)
oracle soft memlock 10485760
oracle hard memlock 10485760
很是納悶為什麼配置都是正確,但是Oracle就是不用標準大頁,於是重啓一下Oracle實例,在告警日誌中發現了蛛絲馬跡,如下所示:
**********************************************************************
2025-08-08T13:50:16.662256+08:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
2025-08-08T13:50:16.662285+08:00
Domain name: system.slice/bpsdbsvr.service
2025-08-08T13:50:16.662302+08:00
Per process system memlock (soft) limit = 64K
2025-08-08T13:50:16.662318+08:00
Expected per process system memlock (soft) limit to lock
instance MAX SHARED GLOBAL AREA (SGA) into memory: 4066M
2025-08-08T13:50:16.662356+08:00
Available system pagesizes:
4K, 2048K
2025-08-08T13:50:16.662387+08:00
Supported system pagesize(s):
2025-08-08T13:50:16.662404+08:00
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
2025-08-08T13:50:16.662421+08:00
4K Configured 11 1040395 NONE
2025-08-08T13:50:16.662450+08:00
2048K 2034 2033 0 NONE
2025-08-08T13:50:16.662466+08:00
RECOMMENDATION:
2025-08-08T13:50:16.662483+08:00
1. Increase per process memlock (soft) limit to at least 4066MB
to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory
2025-08-08T13:50:16.662514+08:00
**********************************************************************
檢查Oracle進程的限制,發現進程的Max locked memory為65536,也就是64K.
# oracle_pid=$(pgrep -f "_pmon_")
# cat /proc/$oracle_pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 33554432 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 46635 46635 processes
Max open files 262144 262144 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 46635 46635 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
也就是説systemd服務啓動Oracle實例時,由於某些原因memlock依然是64K,即日誌中的提示"Per process system memlock (soft) limit = 64K"
於是改用手工啓動數據庫實例,檢查發現標準大頁被Oracle使用了,但是systemd服務啓動Oracle實例就會出現上面標準大頁不被使用的情況
$ grep HugePages /proc/meminfo
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
FileHugePages: 0 kB
HugePages_Total: 2034
HugePages_Free: 4
HugePages_Rsvd: 3
HugePages_Surp: 0
後面和同事查資料,發現systemctl啓動的服務默認不讀取資源限制配置文件(limits.conf). limits.conf中的限制是針對用户會話級別的資源控制，
由PAM模塊在用户登錄時生效。而systemd服務是通過systemd進程直接啓動的，屬於非登錄會話，默認不會觸發PAM的pam_limits.so模塊，
因此/etc/security/limits.conf 中為oracle用户設置的所有資源限制都不會自動應用到通過 systemd 服務啓動的進程.
解決方案
如果想讓systemd服務啓動Oracle實例時memlock限制生效，可以在oracle.service 中直接配置 memlock 限制或者通過PAM讓服務讀取limits.conf.
網上資料推薦直接在systemd服務文件中配置(這是 systemd 推薦的方式，比依賴limits.conf更可靠), 如下所示:
oracle.service原始的配置
[Unit]
Description=Oracle Database Service
After=network.target
[Service]
Type=forking
User=oracle
Group=oinstall
ExecStart=/home/oracle/xxxx/ora19c.sh start
ExecStop=/home/oracle/xxxx/ora19c.sh shutdown
StandardOutput=append:/var/log/rhcs_resource_logs/xxx/xxx.log
RemainAfterExit=yes
KillMode=none
[Install]
WantedBy=multi-user.target
注意: oracle.service配置做了一點混淆,不影響大家理解.
oracle.service修改後配置
[Unit]
Description=Oracle Database Service
After=network.target
[Service]
Type=forking
User=oracle
Group=oinstall
ExecStart=/home/oracle/xxxx/ora19c.sh start
ExecStop=/home/oracle/xxxx/ora19c.sh shutdown
StandardOutput=append:/var/log/rhcs_resource_logs/xxx/xxx.log
RemainAfterExit=yes
KillMode=none
# oracle /etc/security/limits.conf
LimitNPROC=16384
LimitNOFILE=65536
LimitSTACK=10485760
LimitMEMLOCK=10737418240
[Install]
WantedBy=multi-user.target
這樣配置後,通過systemd服務啓動Oracle實例就能正常使用標準大頁了.問題Troubleshooting完美解決. 究其原因還是因為對Linux的systemd服務瞭解得不夠多.不夠深入!
