Feature/3.2 beta7 rest token (#12030)

* add ParameterTypesComparator  for metadataMap

* add ASF for new file

* Fix conflict

* Fix header

* fix import

* fix import

* fix import

* fix import

* fix import

* Fix json

* revert some change

* Remove jsonfactory

* Remove jsonfactory

* Simplify rest client

* Fix protocol

* fix codec

* tmp disable test

* rest metadata resolver add  interface judge

* add rest metadata  resolve unit test & fix  AbstractServiceRestMetadataResolver

* org.apache.dubbo.metadata.rest.ParamType null exclude & add default accept header

* RestProtocolTest

refer add context path
unit test

* some fix

* ADD TODO

* RESOLVE HTTP client java.net.SocketException: socket closed

* RESOLVE HTTP client java.net.SocketException: socket closed

* add spring mvc rest protocol unit test

* rest protocol http response code deal

* rest protocol http response message

* fix some review advice

* fix some review advice

* add rest metadata  resolve unit test & fix  AbstractServiceRestMetadataResolver

* org.apache.dubbo.metadata.rest.ParamType null exclude & add default accept header

* RestProtocolTest

refer add context path
unit test

* some fix

* fix some review advice

* add spring mvc rest protocol unit test

* rest protocol http response code deal

* rest protocol http response message

* URLConnectionRestClient getMessage

* remove unused import

* import fix

* code merge

* remove unused import

* code merge

* code merge

* some fix

* code merge

* code merge

* code merge

* code merge

* Rest http server

* rest protocol do export

* code merge

* rest protocol provider

* rest protocol provider

* rest protocol  response

* rest protocol  netty request

* Fix conflicts

* change  restResult InputStream to bytes protect from  fd leak

* merge code

* Fix import

* Fix import

* Fix uts

* Remove unused code

* Fix logger

* Update okhttp version

* Update okhttp version

* rest protocol  add AnotherUserRestService service ut

* rest protocol  add String & byteArray codec

* Fix version

* Fix uts

* rest protocol  add XMLCodec

* stream release

* ServiceRestMetadata port change int to Integer

* service RestMetadata service map init

* xml codec change  for xxe

* Fix import

* code style

* code style

* Fix shade

* change for rest client recreate & destroy bugs

* remove recreate double check for code merge

* rest client destroy check

* add todo

* consumer merge code

* rest response

* BodyProviderParamParser bytes

* merge upstream 3.2 to 3.2_consumer_proxy_invocation_handler

* merge upstream 3.2 to feature/3.2-rest_protocol_provider

* fix rest provider ut

* path compare

* path compare

* fix provider  rest ut

* fix provider  rest ut

* fix provider  rest ut

* testJaxrsPathPattern ut

* JAXRSServiceRestMetadataResolverTest  ut

* JAXRSServiceRestMetadataResolverTest  ut

* add netty http server

* add netty http server response encode

* rest netty  http server  codec

* rest netty  http server  codec

* some fix  netty http server

* add spring mvc ut  for netty http server

* okhttp_version to 3.14.2 to resolve dependency conflict

* Fix license

* add rest protocol exception mapper

* add TODO ExceptionMapper should be static or instance

* add TODO OKHttpRestClient implements of version >=4.0

* fix licence format

* fix ut

* fix testPathMatcher

* url context path

* format context path from url

* fix illegal logger method invocations:

* remove org.jetbrains.annotations.NotNull

* add Newly created SPI interface  to dubbo-all

* some  format

* merge 3.2

* merge 3.2

* merge 3.2

* Revert "merge 3.2"

This reverts commit ad19804437.

* merge upstream 3.2

* change HttpHandler throws   IOException

* deal with  RPCInvocation build error

* request is present

* remove  request facade unused code

* remove   thirdpart web  server code

* remove   thirdpart web  server code

* remove   unused imports

* remove   unused code

* rest protocol directly implement AbstractProtocol

* code style

* change method  modifier

* form body  java bean convert

* change header name  to RestHeaderEnum

* attachment header

* http handler error info

* rest header distinguish from attachment

* some log & double path check

* remove RestMethodMetadata ServiceRestMetadata refer

* reconstruct http handler

* content-type judge

* add RestInvoker

* setTargetServiceUniqueName

* path mapper & http handler

* remove unused code

* netty thread pool

* RestHttpRequestDecoder @ChannelHandler.Sharable

* add rest client ut

* fix metadata resolver test

* fix metadata resolver test

* add some description to code

* add error test

* error message append

* add request address

* request address

* http netty server reconstruct

* http netty server TODO add SslServerTlsHandler

* replace all pair use

* pair fix

* add NumberUtils ut

* add hex number  ut

* add no annotation primitive ut

* remove unsed code

* throw exception

* add ut

* rename constants & remove unused code

* add ut

* add ut

* add ut

* add ut

* remove ut

* add restClient ut

* add restClient ut

* add DataParseUtils ut

* add path matcher ut

* add request template ut

* add request template ut

* fix xml codec

* some fix

* Class to Class<?>

* media type judge fix and add codec ut

* remove print

* add header map param ut

* add header map param null ut

* MediaTypeUtil ut

* add param & header & multimap     ut

* remove unused import

* fix error log format

* test header param

* test header number  param

* add MediaType ut

* add MediaType & metadata resolve ut

* imports

* add netty request facade ut

* some  format

* add ExceptionMapperTest

* add hasExceptionMapper pre judge

* fix exception mapper throws

* change  static ExceptionMapper to instance

* resolve Java_Zulu_jdk/17.0.6-10/x64 param is not throwable , exception
mapper ut

* fix reflects util some detail

* remove unused code

* add null arg ut

* rest invoker response future complete Exceptionally

* fix http netty server executor

* set rpc context request & response

* add rest token attachment

* add http method ut

* remove unused code

* some fix

* remove keep-alive header

* remove unused code

* close channel

* add response connection:close head

---------

Co-authored-by: suncr <suncairong@moresec.cn>
Co-authored-by: Albumen Kevin <jhq0812@gmail.com>
This commit is contained in:
suncairong163 2023-04-07 09:44:55 +08:00 committed by GitHub
parent e3053618dd
commit 6ac8c83e51
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 255 additions and 24 deletions

View File

@ -23,11 +23,19 @@ import org.apache.dubbo.remoting.http.config.HttpClientConfig;
import org.apache.commons.io.IOUtils;
import org.apache.http.Header;
import org.apache.http.HttpEntityEnclosingRequest;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpDelete;
import org.apache.http.client.methods.HttpEntityEnclosingRequestBase;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpHead;
import org.apache.http.client.methods.HttpOptions;
import org.apache.http.client.methods.HttpPatch;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.methods.HttpTrace;
import org.apache.http.entity.ByteArrayEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
@ -58,14 +66,13 @@ public class HttpClientRestClient implements RestClient {
HttpRequestBase httpRequest = null;
String httpMethod = requestTemplate.getHttpMethod();
if ("GET".equals(httpMethod)) {
httpRequest = new HttpGet(requestTemplate.getURL());
} else if ("POST".equals(httpMethod)) {
HttpPost httpPost = new HttpPost(requestTemplate.getURL());
httpPost.setEntity(new ByteArrayEntity(requestTemplate.getSerializedBody()));
httpRequest = httpPost;
httpRequest = createHttpUriRequest(httpMethod, requestTemplate);
if (httpRequest instanceof HttpEntityEnclosingRequest) {
((HttpEntityEnclosingRequestBase) httpRequest).setEntity(new ByteArrayEntity(requestTemplate.getSerializedBody()));
}
Map<String, Collection<String>> allHeaders = requestTemplate.getAllHeaders();
allHeaders.remove("Content-Length");
@ -91,6 +98,9 @@ public class HttpClientRestClient implements RestClient {
@Override
public byte[] getBody() throws IOException {
if (response.getEntity() == null) {
return new byte[0];
}
return IOUtils.toByteArray(response.getEntity().getContent());
}
@ -151,4 +161,31 @@ public class HttpClientRestClient implements RestClient {
PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
return HttpClients.custom().setConnectionManager(connectionManager).build();
}
protected HttpRequestBase createHttpUriRequest(String httpMethod, RequestTemplate requestTemplate) {
String uri = requestTemplate.getURL();
HttpRequestBase httpUriRequest = null;
if (HttpGet.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpGet(uri);
} else if (HttpHead.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpHead(uri);
} else if (HttpPost.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpPost(uri);
} else if (HttpPut.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpPut(uri);
} else if (HttpPatch.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpPatch(uri);
} else if (HttpDelete.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpDelete(uri);
} else if (HttpOptions.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpOptions(uri);
} else if (HttpTrace.METHOD_NAME.equals(httpMethod)) {
httpUriRequest = new HttpTrace(uri);
} else {
throw new IllegalArgumentException("Invalid HTTP method: " + httpMethod);
}
return httpUriRequest;
}
}

View File

@ -50,10 +50,10 @@ public class URLConnectionRestClient implements RestClient {
HttpURLConnection connection = (HttpURLConnection) new URL(requestTemplate.getURL()).openConnection();
connection.setConnectTimeout(clientConfig.getConnectTimeout());
connection.setReadTimeout(clientConfig.getReadTimeout());
connection.setAllowUserInteraction(false);
connection.setInstanceFollowRedirects(true);
connection.setRequestMethod(requestTemplate.getHttpMethod());
prepareConnection(connection, requestTemplate.getHttpMethod());
// writeHeaders
for (String field : requestTemplate.getAllHeaders().keySet()) {
@ -78,7 +78,6 @@ public class URLConnectionRestClient implements RestClient {
} else {
connection.setChunkedStreamingMode(clientConfig.getChunkLength());
}
connection.setDoOutput(true);
OutputStream out = connection.getOutputStream();
if (gzipEncodedRequest) {
@ -155,4 +154,26 @@ public class URLConnectionRestClient implements RestClient {
};
}
private void prepareConnection(HttpURLConnection connection, String httpMethod) throws IOException {
connection.setDoInput(true);
if ("GET".equals(httpMethod)) {
connection.setInstanceFollowRedirects(true);
} else {
connection.setInstanceFollowRedirects(false);
}
if ("POST".equals(httpMethod) || "PUT".equals(httpMethod) ||
"PATCH".equals(httpMethod) || "DELETE".equals(httpMethod)) {
connection.setDoOutput(true);
} else {
connection.setDoOutput(false);
}
connection.setRequestMethod(httpMethod);
}
}

View File

@ -27,6 +27,7 @@ public enum RestHeaderEnum {
KEEP_ALIVE_HEADER(RestConstant.KEEP_ALIVE_HEADER),
CONNECTION(RestConstant.CONNECTION),
REST_HEADER_PREFIX(RestConstant.REST_HEADER_PREFIX),
TOKEN_KEY(RestConstant.REST_HEADER_PREFIX + RestConstant.TOKEN_KEY),
;

View File

@ -26,7 +26,7 @@ import org.apache.dubbo.rpc.protocol.rest.annotation.consumer.HttpConnectionPreB
import org.apache.dubbo.rpc.protocol.rest.constans.RestConstant;
/**
* add some must attachment
* add some must attachment
*/
@Activate(value = RestConstant.ADD_MUST_ATTTACHMENT,order = 1)
public class AddMustAttachmentIntercept implements HttpConnectionPreBuildIntercept {
@ -41,6 +41,7 @@ public class AddMustAttachmentIntercept implements HttpConnectionPreBuildInterce
requestTemplate.addHeader(RestHeaderEnum.GROUP.getHeader(), serviceRestMetadata.getGroup());
requestTemplate.addHeader(RestHeaderEnum.VERSION.getHeader(), serviceRestMetadata.getVersion());
requestTemplate.addHeader(RestHeaderEnum.PATH.getHeader(), serviceRestMetadata.getServiceInterface());
requestTemplate.addHeader(RestHeaderEnum.TOKEN_KEY.getHeader(), connectionCreateContext.getUrl().getParameter(RestConstant.TOKEN_KEY));
}

View File

@ -17,7 +17,6 @@
package org.apache.dubbo.rpc.protocol.rest.annotation.consumer.inercept;
import org.apache.dubbo.common.URL;
import org.apache.dubbo.common.extension.Activate;
import org.apache.dubbo.metadata.rest.RestMethodMetadata;
import org.apache.dubbo.remoting.http.RequestTemplate;
@ -53,9 +52,10 @@ public class RequestHeaderIntercept implements HttpConnectionPreBuildIntercept {
requestTemplate.addHeader(RestHeaderEnum.ACCEPT.getHeader(), produces);
}
URL url = connectionCreateContext.getUrl();
// URL url = connectionCreateContext.getUrl();
requestTemplate.addKeepAliveHeader(url.getParameter(RestConstant.KEEP_ALIVE_TIMEOUT_PARAM,RestConstant.KEEP_ALIVE_TIMEOUT));
// requestTemplate.addKeepAliveHeader(url.getParameter(RestConstant.KEEP_ALIVE_TIMEOUT_PARAM,RestConstant.KEEP_ALIVE_TIMEOUT));
}

View File

@ -17,11 +17,13 @@
package org.apache.dubbo.rpc.protocol.rest.constans;
import org.apache.dubbo.common.constants.CommonConstants;
import org.apache.dubbo.rpc.Constants;
public interface RestConstant {
String VERSION = CommonConstants.VERSION_KEY;
String GROUP = CommonConstants.GROUP_KEY;
String PATH = CommonConstants.PATH_KEY;
String TOKEN_KEY = Constants.TOKEN_KEY;
String LOCAL_ADDR = "LOCAL_ADDR";
String REMOTE_ADDR = "REMOTE_ADDR";
String LOCAL_PORT = "LOCAL_PORT";

View File

@ -197,11 +197,11 @@ public class NettyHttpResponse implements HttpResponse {
@SuppressWarnings({"rawtypes", "unchecked"})
public static void transformHeaders(NettyHttpResponse nettyResponse, io.netty.handler.codec.http.HttpResponse response) {
if (nettyResponse.isKeepAlive()) {
response.headers().set(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.KEEP_ALIVE);
} else {
response.headers().set(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
}
// if (nettyResponse.isKeepAlive()) {
// response.headers().set(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.KEEP_ALIVE);
// } else {
// response.headers().set(HttpHeaders.Names.CONNECTION, HttpHeaders.Values.CLOSE);
// }
for (Map.Entry<String, List<String>> entry : nettyResponse.getOutputHeaders().entrySet()) {
String key = entry.getKey();

View File

@ -29,6 +29,7 @@ import org.apache.dubbo.common.URL;
import org.apache.dubbo.common.logger.ErrorTypeAwareLogger;
import org.apache.dubbo.common.logger.LoggerFactory;
import org.apache.dubbo.common.threadpool.ThreadPool;
import org.apache.dubbo.rpc.protocol.rest.RestHeaderEnum;
import org.apache.dubbo.rpc.protocol.rest.handler.NettyHttpHandler;
import org.apache.dubbo.rpc.protocol.rest.request.NettyRequestFacade;
@ -61,14 +62,16 @@ public class RestHttpRequestDecoder extends MessageToMessageDecoder<io.netty.han
} catch (IOException e) {
logger.error("", e.getCause().getMessage(), "dubbo rest rest http request handler error", e.getMessage(), e);
} finally {
// write response
try {
nettyHttpResponse.addOutputHeaders(RestHeaderEnum.CONNECTION.getHeader(), "close");
nettyHttpResponse.finish();
} catch (IOException e) {
logger.error("", e.getCause().getMessage(), "dubbo rest rest http response flush error", e.getMessage(), e);
}
}
// write response
try {
nettyHttpResponse.finish();
} catch (IOException e) {
logger.error("", e.getCause().getMessage(), "dubbo rest rest http response flush error", e.getMessage(), e);
}
});

View File

@ -37,11 +37,15 @@ import org.apache.dubbo.rpc.model.ProviderModel;
import org.apache.dubbo.rpc.model.ServiceDescriptor;
import org.apache.dubbo.rpc.protocol.rest.annotation.metadata.MetadataResolver;
import org.apache.dubbo.rpc.protocol.rest.constans.RestConstant;
import org.apache.dubbo.rpc.protocol.rest.exception.DoublePathCheckException;
import org.apache.dubbo.rpc.protocol.rest.exception.mapper.ExceptionHandler;
import org.apache.dubbo.rpc.protocol.rest.exception.mapper.ExceptionMapper;
import org.apache.dubbo.rpc.protocol.rest.rest.AnotherUserRestService;
import org.apache.dubbo.rpc.protocol.rest.rest.AnotherUserRestServiceImpl;
import org.apache.dubbo.rpc.protocol.rest.rest.HttpMethodService;
import org.apache.dubbo.rpc.protocol.rest.rest.HttpMethodServiceImpl;
import org.apache.dubbo.rpc.protocol.rest.rest.RestDemoForTestException;
import org.hamcrest.CoreMatchers;
@ -502,6 +506,53 @@ class JaxrsRestProtocolTest {
exporter.unexport();
}
@Test
void testToken() {
DemoService server = new DemoServiceImpl();
URL url = this.registerProvider(exportUrl, server, DemoService.class);
URL nettyUrl = url.addParameter(RestConstant.TOKEN_KEY, "TOKEN");
Exporter<DemoService> exporter = protocol.export(proxy.getInvoker(server, DemoService.class, nettyUrl));
DemoService demoService = this.proxy.getProxy(protocol.refer(DemoService.class, nettyUrl));
Assertions.assertEquals("Hello, hello", demoService.sayHello("hello"));
exporter.unexport();
}
@Test
void testHttpMethods() {
testHttpMethod(org.apache.dubbo.remoting.Constants.OK_HTTP);
testHttpMethod(org.apache.dubbo.remoting.Constants.APACHE_HTTP_CLIENT);
testHttpMethod(org.apache.dubbo.remoting.Constants.URL_CONNECTION);
}
void testHttpMethod(String restClient) {
HttpMethodService server = new HttpMethodServiceImpl();
URL url = URL.valueOf("rest://127.0.0.1:" + NetUtils.getAvailablePort()
+ "/?version=1.0.0&interface=org.apache.dubbo.rpc.protocol.rest.rest.HttpMethodService&"
+ org.apache.dubbo.remoting.Constants.CLIENT_KEY + "=" + restClient);
url = this.registerProvider(url, server, HttpMethodService.class);
Exporter<HttpMethodService> exporter = protocol.export(proxy.getInvoker(server, HttpMethodService.class, url));
HttpMethodService demoService = this.proxy.getProxy(protocol.refer(HttpMethodService.class, url));
String expect = "hello";
Assertions.assertEquals(null, demoService.sayHelloHead());
Assertions.assertEquals(expect, demoService.sayHelloDelete("hello"));
Assertions.assertEquals(expect, demoService.sayHelloGet("hello"));
Assertions.assertEquals(expect, demoService.sayHelloOptions("hello"));
// Assertions.assertEquals(expect, demoService.sayHelloPatch("hello"));
Assertions.assertEquals(expect, demoService.sayHelloPost("hello"));
Assertions.assertEquals(expect, demoService.sayHelloPut("hello"));
exporter.unexport();
}
public static class TestExceptionMapper implements ExceptionHandler<RuntimeException> {
@Override

View File

@ -0,0 +1,59 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dubbo.rpc.protocol.rest.rest;
import javax.ws.rs.*;
@Path("/demoService")
public interface HttpMethodService {
@POST
@Path("/sayPost")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloPost(@QueryParam("name") String name);
@DELETE
@Path("/sayDelete")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloDelete(@QueryParam("name") String name);
@HEAD
@Path("/sayHead")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloHead();
@GET
@Path("/sayGet")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloGet(@QueryParam("name") String name);
@PUT
@Path("/sayPut")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloPut(@QueryParam("name") String name);
@PATCH
@Path("/sayPatch")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloPatch(@QueryParam("name") String name);
@OPTIONS
@Path("/sayOptions")
@Consumes({javax.ws.rs.core.MediaType.TEXT_PLAIN})
String sayHelloOptions(@QueryParam("name") String name);
}

View File

@ -0,0 +1,56 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dubbo.rpc.protocol.rest.rest;
public class HttpMethodServiceImpl implements HttpMethodService{
@Override
public String sayHelloPost(String name) {
return name;
}
@Override
public String sayHelloDelete(String name) {
return name;
}
@Override
public String sayHelloHead() {
return "hello";
}
@Override
public String sayHelloGet(String name) {
return name;
}
@Override
public String sayHelloPut(String name) {
return name;
}
@Override
public String sayHelloPatch(String name) {
return name;
}
@Override
public String sayHelloOptions(String name) {
return name;
}
}