NoaSoftware
/
TencentOS-kernel
forked from TencentOS/TencentOS-kernel
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
TencentOS-kernel/arch/x86/crypto
History
Ard Biesheuvel d8516e9732 crypto: x86/aes-ni-xts - use direct calls to and 4-way stride 
commit 86ad60a65f29dd862a11c22bb4b5be28d6c5cef1 upstream.

The XTS asm helper arrangement is a bit odd: the 8-way stride helper
consists of back-to-back calls to the 4-way core transforms, which
are called indirectly, based on a boolean that indicates whether we
are performing encryption or decryption.

Given how costly indirect calls are on x86, let's switch to direct
calls, and given how the 8-way stride doesn't really add anything
substantial, use a 4-way stride instead, and make the asm core
routine deal with any multiple of 4 blocks. Since 512 byte sectors
or 4 KB blocks are the typical quantities XTS operates on, increase
the stride exported to the glue helper to 512 bytes as well.

As a result, the number of indirect calls is reduced from 3 per 64 bytes
of in/output to 1 per 512 bytes of in/output, which produces a 65% speedup
when operating on 1 KB blocks (measured on a Intel(R) Core(TM) i7-8650U CPU)

Fixes: 9697fa39efd3f ("x86/retpoline/crypto: Convert crypto assembler indirect jumps")
Tested-by: Eric Biggers <ebiggers@google.com> # x86_64
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
[ardb: rebase onto stable/linux-5.4.y]
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
 		2021-04-12 12:53:14 +08:00
..
Makefile Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
aegis128-aesni-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
aegis128-aesni-glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
aes_ctrby8_avx-x86_64.S crypto: aesni - add compatibility with IAS 2021-03-16 16:33:48 +08:00
aes_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
aesni-intel_asm.S crypto: x86/aes-ni-xts - use direct calls to and 4-way stride 2021-04-12 12:53:14 +08:00
aesni-intel_avx-x86_64.S crypto: aesni - Use TEST %reg,%reg instead of CMP $0,%reg 2021-04-12 12:53:14 +08:00
aesni-intel_glue.c crypto: x86/aes-ni-xts - use direct calls to and 4-way stride 2021-04-12 12:53:14 +08:00
blowfish-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
blowfish_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
camellia-aesni-avx-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
camellia-aesni-avx2-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
camellia-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
camellia_aesni_avx2_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
camellia_aesni_avx_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
camellia_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
cast5-avx-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
cast5_avx_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
cast6-avx-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
cast6_avx_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
chacha-avx2-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
chacha-avx512vl-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
chacha-ssse3-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
chacha_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
crc32-pclmul_asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
crc32-pclmul_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
crc32c-intel_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
crc32c-pcl-intel-asm_64.S crypto: x86/crc32c - fix building with clang ias 2021-03-16 16:39:35 +08:00
crct10dif-pcl-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
crct10dif-pclmul_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
des3_ede-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
des3_ede_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
ghash-clmulni-intel_asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
ghash-clmulni-intel_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
glue_helper-asm-avx.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
glue_helper-asm-avx2.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
glue_helper.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
nh-avx2-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
nh-sse2-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
nhpoly1305-avx2-glue.c crypto: arch/nhpoly1305 - process in explicit 4k chunks 2021-03-16 16:27:38 +08:00
nhpoly1305-sse2-glue.c crypto: arch/nhpoly1305 - process in explicit 4k chunks 2021-03-16 16:27:38 +08:00
poly1305-avx2-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
poly1305-sse2-x86_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
poly1305_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
serpent-avx-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
serpent-avx2-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
serpent-sse2-i586-asm_32.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
serpent-sse2-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
serpent_avx2_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
serpent_avx_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
serpent_sse2_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
sha1_avx2_x86_64_asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha1_ni_asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha1_ssse3_asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha1_ssse3_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha256-avx-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha256-avx2-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha256-ssse3-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha256_ni_asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha256_ssse3_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha512-avx-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha512-avx2-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha512-ssse3-asm.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
sha512_ssse3_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
twofish-avx-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
twofish-i586-asm_32.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
twofish-x86_64-asm_64-3way.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
twofish-x86_64-asm_64.S Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
twofish_avx_glue.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00
twofish_glue.c Init Repo base on linux 5.4.32 long term, and add base tlinux kernel interfaces. 2021-03-16 11:01:34 +08:00
twofish_glue_3way.c crypto: x86 - Regularize glue function prototypes 2021-04-12 12:53:14 +08:00