NoaSoftware
/
TencentOS-kernel
forked from TencentOS/TencentOS-kernel
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
9,480 Commits 59 Branches 0 Tags 1.5 GiB
C 97.1%
Assembly 1.6%
C++ 0.6%
Makefile 0.3%
Perl 0.1%
Go to file
Hyeonggon Yoo a0507babe2 mm, slub: use prefetchw instead of prefetch 
commit 04b4b006139b58ffbd90df3befecc13711b1faf8 upstream.

Commit 0ad9500e16fe ("slub: prefetch next freelist pointer in
slab_alloc()") introduced prefetch_freepointer() because when other
cpu(s) freed objects into a page that current cpu owns, the freelist
link is hot on cpu(s) which freed objects and possibly very cold on
current cpu.

But if freelist link chain is hot on cpu(s) which freed objects, it's
better to invalidate that chain because they're not going to access
again within a short time.

So use prefetchw instead of prefetch.  On supported architectures like
x86 and arm, it invalidates other copied instances of a cache line when
prefetching it.

Before:

Time: 91.677

 Performance counter stats for 'hackbench -g 100 -l 10000':
        1462938.07 msec cpu-clock                 #   15.908 CPUs utilized
          18072550      context-switches          #   12.354 K/sec
           1018814      cpu-migrations            #  696.416 /sec
            104558      page-faults               #   71.471 /sec
     1580035699271      cycles                    #    1.080 GHz                      (54.51%)
     2003670016013      instructions              #    1.27  insn per cycle           (54.31%)
        5702204863      branch-misses                                                 (54.28%)
      643368500985      cache-references          #  439.778 M/sec                    (54.26%)
       18475582235      cache-misses              #    2.872 % of all cache refs      (54.28%)
      642206796636      L1-dcache-loads           #  438.984 M/sec                    (46.87%)
       18215813147      L1-dcache-load-misses     #    2.84% of all L1-dcache accesses  (46.83%)
      653842996501      dTLB-loads                #  446.938 M/sec                    (46.63%)
        3227179675      dTLB-load-misses          #    0.49% of all dTLB cache accesses  (46.85%)
      537531951350      iTLB-loads                #  367.433 M/sec                    (54.33%)
         114750630      iTLB-load-misses          #    0.02% of all iTLB cache accesses  (54.37%)
      630135543177      L1-icache-loads           #  430.733 M/sec                    (46.80%)
       22923237620      L1-icache-load-misses     #    3.64% of all L1-icache accesses  (46.76%)

      91.964452802 seconds time elapsed

      43.416742000 seconds user
    1422.441123000 seconds sys

After:

Time: 90.220

 Performance counter stats for 'hackbench -g 100 -l 10000':
        1437418.48 msec cpu-clock                 #   15.880 CPUs utilized
          17694068      context-switches          #   12.310 K/sec
            958257      cpu-migrations            #  666.651 /sec
            100604      page-faults               #   69.989 /sec
     1583259429428      cycles                    #    1.101 GHz                      (54.57%)
     2004002484935      instructions              #    1.27  insn per cycle           (54.37%)
        5594202389      branch-misses                                                 (54.36%)
      643113574524      cache-references          #  447.409 M/sec                    (54.39%)
       18233791870      cache-misses              #    2.835 % of all cache refs      (54.37%)
      640205852062      L1-dcache-loads           #  445.386 M/sec                    (46.75%)
       17968160377      L1-dcache-load-misses     #    2.81% of all L1-dcache accesses  (46.79%)
      651747432274      dTLB-loads                #  453.415 M/sec                    (46.59%)
        3127124271      dTLB-load-misses          #    0.48% of all dTLB cache accesses  (46.75%)
      535395273064      iTLB-loads                #  372.470 M/sec                    (54.38%)
         113500056      iTLB-load-misses          #    0.02% of all iTLB cache accesses  (54.35%)
      628871845924      L1-icache-loads           #  437.501 M/sec                    (46.80%)
       22585641203      L1-icache-load-misses     #    3.59% of all L1-icache accesses  (46.79%)

      90.514819303 seconds time elapsed

      43.877656000 seconds user
    1397.176001000 seconds sys

[xuyu]:
The latency of 'hackbench -g 100 -l 10000' in a x86 guest with 8 CPU
and 16G memory is 92.81 (w/o this path) and 91.2018 (w/ this patch),
respectively.

Link: https://lkml.org/lkml/2021/10/8/598=20
Link: https://lkml.kernel.org/r/20211011144331.70084-1-42.hyeyoo@gmail.com
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: Christoph Lameter <cl@linux.com>
Cc: Pekka Enberg <penberg@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Xu Yu <xuyu@linux.alibaba.com>
Reviewed-by: Gang Deng <gavin.dg@linux.alibaba.com>
Signed-off-by: johnnyaiai <johnnyaiai@tencent.com>
 		2022-11-03 10:49:01 +08:00
Documentation docs/bpf: Add BPF ring buffer design notes 2022-08-30 11:33:29 +08:00
LICENSES Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
arch dist: tk4: fix debugedit compact issue 2022-10-17 17:31:28 +08:00
block Add sysfs attribute to hide disk devices 2022-07-25 16:24:47 +08:00
certs certs: Fix blacklist flag type confusion 2021-04-12 12:52:25 +08:00
crypto crypto: rng - fix crypto_rng_reset() refcounting when !CRYPTO_STATS 2021-05-20 16:07:34 +08:00
dist dist: fix remote name in check-commits 2022-10-18 11:37:20 +08:00
drivers virtio-net: use NETIF_F_GRO_HW instead of NETIF_F_LRO 2022-10-18 10:47:43 +08:00
fs bpf: Refactor to provide aux info to bpf_iter_init_seq_priv_t 2022-08-30 11:33:28 +08:00
include Revert "mm/lru: introduce TestClearPageLRU" 2022-09-05 20:56:40 +08:00
init bpf: Introduce BPF_PROG_TYPE_LSM 2022-08-30 11:33:21 +08:00
ipc ipc/util.c: sysvipc_find_ipc() incorrectly updates position index 2021-03-16 16:27:51 +08:00
kernel sli: fix the compiling error caused by memory access out of range 2022-10-14 16:20:22 +08:00
lib bug: Remove redundant condition check in report_bug 2021-05-20 16:07:55 +08:00
mm mm, slub: use prefetchw instead of prefetch 2022-11-03 10:49:01 +08:00
net can: bcm: delay release of struct bcm_op after synchronize_rcu() 2022-09-27 10:29:35 +08:00
package open CONFIG_JUMP_LABEL to optimize very unlikely/likely branches 2022-11-03 10:48:47 +08:00
samples kfifo: fix ternary sign extension bugs 2021-05-20 16:08:01 +08:00
scripts bpf: fix selftest case compile error for bpf_timer 2022-08-30 11:33:34 +08:00
security bpf: Implement bpf_local_storage for inodes 2022-08-30 11:33:29 +08:00
sound ALSA: usb: midi: don't return -ENOMEM when usb_urb_ep_type_check fails 2021-05-20 16:07:59 +08:00
tools bpf, selftests: Add redirect_peer selftest 2022-08-30 11:33:35 +08:00
usr Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
virt KVM: Stop looking for coalesced MMIO zones if the bus is destroyed 2021-05-20 16:07:44 +08:00
.clang-format Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
.cocciconfig Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
.get_maintainer.ignore Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
.gitattributes dist: hook into kernel build system 2022-10-17 02:25:35 +08:00
.gitignore dist: hook into kernel build system 2022-10-17 02:25:35 +08:00
COPYING Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
CREDITS Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
Kbuild Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
Kconfig Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
MAINTAINERS bpf: Introduce BPF_PROG_TYPE_LSM 2022-08-30 11:33:21 +08:00
Makefile dist: hook into kernel build system 2022-10-17 02:25:35 +08:00
README Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
README.md Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
backport_remove_lists.txt Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
tools_key.pub Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00

README.md

Tencent Linux Kernel 4.0