Ni Xun ced67c8df7 mm: allow a controlled amount of unfairness in the page lock ... upstream commit id: 5ef64cc8987a9211d3f3667331ba3411a94ddc79 Commit 2a9127fcf229 ("mm: rewrite wait_on_page_bit_common() logic") made the page locking entirely fair, in that if a waiter came in while the lock was held, the lock would be transferred to the lockers strictly in order. That was intended to finally get rid of the long-reported watchdog failures that involved the page lock under extreme load, where a process could end up waiting essentially forever, as other page lockers stole the lock from under it. It also improved some benchmarks, but it ended up causing huge performance regressions on others, simply because fair lock behavior doesn't end up giving out the lock as aggressively, causing better worst-case latency, but potentially much worse average latencies and throughput. Instead of reverting that change entirely, this introduces a controlled amount of unfairness, with a sysctl knob to tune it if somebody needs to. But the default value should hopefully be good for any normal load, allowing a few rounds of lock stealing, but enforcing the strict ordering before the lock has been stolen too many times. There is also a hint from Matthieu Baerts that the fair page coloring may end up exposing an ABBA deadlock that is hidden by the usual optimistic lock stealing, and while the unfairness doesn't fix the fundamental issue (and I'm still looking at that), it avoids it in practice. The amount of unfairness can be modified by writing a new value to the 'sysctl_page_lock_unfairness' variable (default value of 5, exposed through /proc/sys/vm/page_lock_unfairness), but that is hopefully something we'd use mainly for debugging rather than being necessary for any deep system tuning. This whole issue has exposed just how critical the page lock can be, and how contended it gets under certain locks. And the main contention doesn't really seem to be anything related to IO (which was the origin of this lock), but for things like just verifying that the page file mapping is stable while faulting in the page into a page table. Link: https://lore.kernel.org/linux-fsdevel/ed8442fd-6f54-dd84-cd4a-941e8b7ee603@MichaelLarabel.com/ Link: https://www.phoronix.com/scan.php?page=article&item=linux-50-59&num=1 Link: https://lore.kernel.org/linux-fsdevel/c560a38d-8313-51fb-b1ec-e904bd8836bc@tessares.net/ Reported-and-tested-by: Michael Larabel <Michael@michaellarabel.com> Tested-by: Matthieu Baerts <matthieu.baerts@tessares.net> Cc: Dave Chinner <david@fromorbit.com> Cc: Matthew Wilcox <willy@infradead.org> Cc: Chris Mason <clm@fb.com> Cc: Jan Kara <jack@suse.cz> Cc: Amir Goldstein <amir73il@gmail.com> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Ni Xun <richardni@tencent.com>		2022-11-03 11:38:02 +08:00
..
bpf	bpf: disable 'optimize bpf_tail_call' for now	2022-08-30 11:33:36 +08:00
cgroup	sli: fix the compiling error caused by memory access out of range	2022-10-14 16:20:22 +08:00
configs	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
debug	kdb: Make memory allocations more robust	2021-04-12 12:52:10 +08:00
dma	swiotlb: fix "x86: Don't panic if can not alloc buffer for swiotlb"	2021-03-16 16:40:57 +08:00
events	perf: Expose get/put_callchain_entry()	2022-08-30 11:33:26 +08:00
gcov	gcov: re-fix clang-11+ support	2021-05-20 16:06:55 +08:00
irq	genirq/matrix: Prevent allocation counter corruption	2021-05-20 16:07:24 +08:00
livepatch	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
locking	locking/qspinlock: Introduce llc aware spinlock	2022-03-02 10:48:05 +08:00
power	PM: EM: postpone creating the debugfs dir till fs_initcall	2021-04-12 12:53:30 +08:00
printk	printk: fix deadlock when kernel panic	2021-04-12 12:52:39 +08:00
rcu	rcu/nocb: Perform deferred wake up before last idle's need_resched() check	2021-04-12 12:52:37 +08:00
sched	list: add "list_del_init_careful()" to go with "list_empty_careful()"	2022-11-03 11:35:29 +08:00
time	posix-timers: Preserve return value in clock_adjtime32()	2021-05-20 16:07:22 +08:00
tkernel	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
trace	selftests/bpf: sync fentry/fexit test from upstream	2022-08-30 11:33:31 +08:00
.gitignore	kbuild: update config_data.gz only when the content of .config is changed	2021-05-20 16:07:35 +08:00
Kconfig.freezer	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
Kconfig.hz	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
Kconfig.locks	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
Kconfig.preempt	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
Makefile	kbuild: update config_data.gz only when the content of .config is changed	2021-05-20 16:07:35 +08:00
acct.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
async.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
audit.c	audit: fix a net reference leak in audit_list_rules_send()	2021-03-16 16:29:25 +08:00
audit.h	audit: fix a net reference leak in audit_list_rules_send()	2021-03-16 16:29:25 +08:00
audit_fsnotify.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
audit_tree.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
audit_watch.c	audit: CONFIG_CHANGE don't log internal bookkeeping as an event	2021-03-16 16:36:51 +08:00
auditfilter.c	audit: fix a net reference leak in audit_list_rules_send()	2021-03-16 16:29:25 +08:00
auditsc.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
backtracetest.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
bounds.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
capability.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
compat.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
configs.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
context_tracking.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
cpu.c	kernel/cpu: add arch override for clear_tasks_mm_cpumask() mm handling	2021-03-16 16:42:28 +08:00
cpu_pm.c	kernel/cpu_pm: Fix uninitted local in cpu_pm	2021-03-16 16:29:48 +08:00
crash_core.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
crash_dump.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
cred.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
delayacct.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
dma.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
exec_domain.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
exit.c	don't dump the threads that had been already exiting when zapped.	2021-03-16 16:40:57 +08:00
extable.c	bpf: Add support for BTF pointers to x86 JIT	2022-08-30 11:33:17 +08:00
fail_function.c	fail_function: Remove a redundant mutex unlock	2021-03-16 16:41:23 +08:00
fork.c	exec: Transform exec_update_mutex into a rw_semaphore	2021-04-12 12:51:05 +08:00
freezer.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
futex.c	Revert 337f13046ff0 ("futex: Allow FUTEX_CLOCK_REALTIME with FUTEX_WAIT op")	2021-05-20 16:07:35 +08:00
gen_kheaders.sh	kbuild: add variables for compression tools	2021-03-16 16:35:19 +08:00
groups.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
hung_task.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
iomem.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
irq_work.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
jump_label.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kallsyms.c	kallsyms: Refactor kallsyms_show_value() to take cred	2021-03-16 16:31:37 +08:00
kcmp.c	exec: Transform exec_update_mutex into a rw_semaphore	2021-04-12 12:51:05 +08:00
kcov.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kexec.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kexec_core.c	kernel: kexec: remove the lock operation of system_transition_mutex	2021-04-12 12:51:43 +08:00
kexec_elf.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kexec_file.c	ima: Free IMA measurement buffer after kexec syscall	2021-04-12 12:52:21 +08:00
kexec_internal.h	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kheaders.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kmod.c	kmod: make request_module() return an error when autoloading is disabled	2021-03-16 16:26:15 +08:00
kprobes.c	tracing/kprobe: Fix to support kretprobe events on unloaded modules	2021-04-12 12:51:58 +08:00
ksysfs.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
kthread.c	kthread: Fix PF_KTHREAD vs to_kthread() race	2022-03-02 10:48:07 +08:00
latencytop.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
module-internal.h	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
module.c	kabi: check modver major version and features	2021-06-23 07:24:18 +00:00
module_signature.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
module_signing.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
notifier.c	kernel/notifier.c: intercept duplicate registrations to avoid infinite loops	2021-03-16 16:36:43 +08:00
nsproxy.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
padata.c	padata: add separate cpuhp node for CPUHP_PADATA_DEAD	2021-03-16 16:28:57 +08:00
panic.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
params.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
pid.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
pid_namespace.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
profile.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
ptrace.c	ptrace: Set PF_SUPERPRIV when checking capability	2021-03-16 16:41:32 +08:00
range.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
reboot.c	reboot: fix overflow parsing reboot cpu number	2021-03-16 16:40:55 +08:00
relay.c	kernel/relay.c: fix memleak on destroy relay channel	2021-03-16 16:34:28 +08:00
resource.c	/dev/mem: Revoke mappings when a driver claims the region	2021-03-16 16:30:25 +08:00
rseq.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
seccomp.c	seccomp: Add missing return in non-void function	2021-04-12 12:52:36 +08:00
signal.c	ptrace: fix task_join_group_stop() for the case when current is traced	2021-03-16 16:40:18 +08:00
smp.c	smp: Fix smp_call_function_single_async prototype	2021-05-20 16:08:01 +08:00
smpboot.c	kthread: Extract KTHREAD_IS_PER_CPU	2021-04-12 12:51:51 +08:00
smpboot.h	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
softirq.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
stackleak.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
stacktrace.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
stop_machine.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
sys.c	kernel/sys.c: avoid copying possible padding bytes in copy_to_user	2021-03-16 16:36:43 +08:00
sys_ni.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
sysctl-test.c	kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec()	2021-03-16 16:36:31 +08:00
sysctl.c	mm: allow a controlled amount of unfairness in the page lock	2022-11-03 11:38:02 +08:00
sysctl_binary.c	net: change TCP_TIMEWAIT_LEN on runtime	2021-03-16 16:43:58 +08:00
task_work.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
taskstats.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
test_kprobes.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
torture.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
tracepoint.c	tracepoint: Do not fail unregistering a probe due to memory failure	2021-04-12 12:52:27 +08:00
tsacct.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
ucount.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
uid16.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
uid16.h	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
umh.c	usermodehelper: reset umask to default before executing user process	2021-03-16 16:37:54 +08:00
up.c	smp: Fix smp_call_function_single_async prototype	2021-05-20 16:08:01 +08:00
user-return-notifier.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
user.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
user_namespace.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
utsname.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
utsname_sysctl.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
watchdog.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
watchdog_hld.c	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00
workqueue.c	workqueue: Move the position of debug_work_activate() in __queue_work()	2021-05-20 16:07:02 +08:00
workqueue_internal.h	Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces.	2021-03-16 11:01:34 +08:00